Back to AI Research

AI Research

InstructMesh: Selective Refinement of Generative 3D... | AI Research

Key Takeaways

  • InstructMesh is an interactive tool designed to help users repair 3D models generated by AI so they can be successfully fabricated.
  • Recent advances in generative AI allow users to create 3D models from text or images.
  • However, these models prioritize visual plausibility over geometric accuracy, often generating results with flaws that compromise their intended use post-fabrication.
  • Users can invoke edit operations via natural language prompts or slider controls.
  • By operating directly on the intermediate latent representation, InstructMesh allows users to apply robust geometric corrections without requiring expert modeling skills.
Paper AbstractExpand

Recent advances in generative AI allow users to create 3D models from text or images. However, these models prioritize visual plausibility over geometric accuracy, often generating results with flaws that compromise their intended use post-fabrication. We present InstructMesh, an interactive post-generation refinement tool that enables selective repair of generative 3D models through region selection and targeted operations, such as opening or sealing voids, or adjusting local thickness. Users can invoke edit operations via natural language prompts or slider controls. By operating directly on the intermediate latent representation, InstructMesh allows users to apply robust geometric corrections without requiring expert modeling skills. To inform our design, we first analyze common fabrication-related failure modes in outputs from state-of-the-art generative tools. We then conduct two user studies, demonstrating that novices can identify and perform fabrication-relevant repairs on generative outputs using InstructMesh, and revealing user preference for hybrid interfaces that combine slider controls with natural language input.

InstructMesh is an interactive tool designed to help users repair 3D models generated by AI so they can be successfully fabricated. While current generative AI models can create visually appealing 3D objects from text or images, they often produce geometric flaws—such as sealed openings, missing connections, or incorrect wall thicknesses—that make the objects unusable for 3D printing. InstructMesh allows users to fix these issues by selecting specific regions of a model and applying targeted geometric corrections without requiring expert 3D modeling skills.

Identifying Fabrication Flaws

To understand why generative 3D models often fail in physical settings, the researchers analyzed 120 popular 3D models from the website Thingiverse. By reconstructing these models using the generative tool Trellis, they identified a recurring set of geometric errors. The most common issues included extraneous artifacts (65.8%), missing openings (48.6%), and hollowing errors (36.9%). The study found that 78.3% of the analyzed models contained more than one type of flaw, suggesting that these errors are a systemic limitation of current image-based generative training.

How InstructMesh Works

InstructMesh operates on the intermediate latent representation of a 3D model rather than the final mesh. When a user identifies a flaw, they highlight the problematic area and provide instructions through either natural language prompts or a slider-based interface. The system then modifies the underlying voxel grid—the "blueprint" used by the AI to build the model—before decoding it into a final, corrected mesh. This approach allows the generative model to maintain the object's overall texture and surface details while incorporating the user's specific geometric changes.

User Control and Visualization

A primary challenge in latent-space editing is that the intermediate representation is not inherently visual, making it difficult for users to predict how an edit will change the final object. To address this, InstructMesh provides a preview visualization. Before committing to a change, users see a color-coded overlay: green indicates additive changes, and red indicates subtractive changes. This "What You See Is What You Get" (WYSIWYG) style of interaction allows users to verify their edits and maintain control over the generative process.

Research Findings

In two user studies involving 12 participants, the researchers found that novices were able to successfully identify and repair fabrication-relevant flaws using the tool. The studies revealed that users preferred a hybrid interface that combined the flexibility of natural language with the precision of slider controls. While the tool effectively addresses static geometric flaws, the authors note that it does not currently support the creation of dynamic mechanisms, such as hinges or interlocking parts, which require more complex mechanical and geometric constraints.

Comments (0)

No comments yet

Be the first to share your thoughts!