MIT Tool Lets Users Repair AI-Generated 3D Models Before Printing

AI-generated 3D models look convincing on screen. They fall apart when you try to print them. A new system from MIT's Computer Science and Artificial Intelligence Laboratory aims to fix that.

What InstructMesh does

InstructMesh pairs Microsoft's TRELLIS 3D generator with GPT-4. A user highlights a region of a generated mesh, describes the defect in plain language, and the system applies a local geometric fix. For smaller adjustments, sliders let you change thickness or extrusion without typing anything.

The CSAIL team tested the system by having TRELLIS recreate popular Thingiverse models. Nearly 80 percent of the generated meshes contained structural flaws that would have caused print failures. Novices using InstructMesh caught and repaired those issues about 90 percent of the time, according to expert review.

Printed results

The researchers printed a range of repaired objects: a mug wrapped by a dragon whose tail forms the handle, butterfly-wing glasses, a blue shell-shaped whistle, an octopus-style dispenser with eight pouring tentacles, a knee brace patterned to match denim, and a shrimp-like bristle-bot enclosure with a motor inside.

Lead author Faraz Faruqi, now at Google, said the goal was to combine the visual strengths of 3D generators with the reasoning of large language models in an interactive space. The result is objects that look like what you asked for and actually work.

Why it matters

Generative AI has made 3D modeling accessible, but most text-to-3D tools optimize for appearance, not manufacturability. That gap has kept a lot of AI-generated designs off print beds. InstructMesh closes it by letting users repair meshes in the same environment where they generate them.

The team will present the work at the ACM Symposium on User Interface Software and Technology in November. The project was supported by Google and the MIT-HPI Collaborative Research Program, with additional contributions from Northeastern University.