InstructMesh lets users repair generated 3D designs before printing

A generated mug can look convincing on a screen and still fail as a cup. InstructMesh addresses that gap by letting users select parts of an AI-generated 3D design and describe the changes they want before fabrication.

Researchers at MIT CSAIL, Google and Northeastern University developed the interface, according to MIT's report on the project. Their examples include household objects and enclosures for small robots. The work focuses on helping a person refine the geometry of a generated object, rather than accepting the generator's first attempt as a finished design.

Local edits connect appearance with function

InstructMesh combines Microsoft's TRELLIS system, which generates 3D models from text and image prompts, with GPT-4. The language model helps interpret a user's description of a problem, while the interface supports changes to the geometry for the user to inspect and approve.

Users can highlight a specific region of a design instead of replacing the entire object. Sliders also give them control over changes such as enlarging or extruding a part. That combination lets a person express both a visual idea and an adjustment to make the object more useful. It still leaves evaluation with the user.

The team's demonstrations include a dragon-shaped mug whose tail becomes the handle, a shell-like whistle and glasses decorated with butterfly wings. An octopus-like dispenser directs liquids through its tentacles into multiple cups. These are examples reported by the project team, not a Franklin test of print quality or durability.

MIT also describes a denim-looking knee brace and a shrimp-shaped bristle bot with a motor inside its enclosure. The brace example should not be read as evidence of medical certification, clinical benefit or suitability for a particular patient. The report establishes a fabrication demonstration.

Novices repaired flaws in a reported study

To examine whether newcomers could identify functional problems, the researchers had TRELLIS recreate popular printable models from Thingiverse. Nearly 80% of the generated models had some structural flaw, according to MIT's account. That proportion applies to this set of recreated models; it is not an established error rate for every text-to-3D tool or every use of TRELLIS.

The team then asked novice users to identify and repair those issues through InstructMesh. MIT reports that they succeeded at both around 90% of the time, as assessed by an expert. Users also made objects resembling phone stands and vases, and reported that the interface was easy to use.

Those observations support the value of an interactive repair step in the described study. They do not show that untrained users can certify arbitrary objects for load-bearing, food-contact or safety-critical uses. A convincing shape and a successful print answer different questions from material performance under a specific load.

The captured report does not give the study's participant count or a complete breakdown of the tested failure types. Those omissions limit how far the headline success rate can be generalized. Readers assessing the work should keep the reported study conditions attached to its numbers.

Physics and augmented reality are proposed next steps

Lead author Faraz Faruqi describes possible future work that would put InstructMesh in an augmented-reality setting, allowing a request to use the surrounding environment as context. MIT also names physics simulation and the newer TRELLIS.2 as possible additions.

Physics simulation could help investigate how a design behaves when dropped or which materials suit a particular use. In this report it remains a possible extension, not an established capability of the demonstrated interface. Likewise, the proposed augmented-reality workflow has not been described as a released product.

The researchers plan to present the work at the ACM Symposium on User Interface Software and Technology in November. The project offers a specific approach to repairing generated geometry: a person identifies the problem, requests a local change and examines the result before printing. Its reported study is encouraging evidence for that interaction, with broader claims about dependable fabrication still requiring their own tests.