AffordCraft: Scalable Construction of Task-Ready Simulation Assets from Single Images

Everyday objects repeat the same few mechanisms. AffordCraft retrieves an asset whose parts and joints already work, fits it to the photo, and keeps it only if it passes a physical test.

1,703/ 2,000

photos become physically valid assets

0.8GPU-min

per accepted asset

11,372

library entries, built once, reused

162/ 237

objects built from cluttered photos

How it works

A kitchen photo, one instruction, six steps.

01 / 06 · Ground

⌘

selector · Qwen3-VL-8Bcategory ✓mechanism ✓subtype matchconfidence 0.95geometry 0.98

X′ = sRX + to′ = sRo + td′ = Rdrevolute limits unchanged

Export

loads in the simulator's format

Inertia

collision shape and valid inertia per body

Settle

comes to rest under gravity, no interpenetration

Support

rests on real contact, no hidden anchor

Joint(door, revolute)

the requested part moves the requested way

PASS

“open the microwave door fully”scripted teacher · passed

01 · Ground

Find the object and the part the task acts on.

The full photo becomes a query: category, part, action and motion.

category Microwavepart dooraction openmotion revolute

02 · Retrieve

Rank the library by appearance.

Every library entry is ranked against the photo by DINOv2.

03 · Select

Let structure overrule the visual rank.

Candidates are checked against their recorded parts and joints; the fourth matches.

04 · Adapt

Fit the entry, keep the mechanism.

One similarity transform; every body and joint is kept.

05 · Gate

Admit it only if physics agrees.

It must export, settle, rest on real support and move at the requested part.

06 · Use

Drop it into a scene and let a robot use it.

Composed into a scene, the microwave door is opened by a robot.

Method

Structure is built once, then reused.

Parts, joints and physics are modeled once in the library and reused for every photo.

Task-conditioned query

The instruction fixes which part must move.

Structure-aware selection

Judged by recorded bodies and joints, not only by looks.

Mechanism-preserving adapter

Every body and joint is imported intact.

Physical feedback

A failure triggers repair and reselection.

G(Â) = Export∧Inertia∧Settle∧Support∧Joint(p, m)

One physical gate scores every method.

Results

More assets that work, for a fraction of the compute.

Same photos, same instruction, same physical gate for every method.

Accepted share vs. cost

Pass rate vs. GPU-minutes per accepted asset.

Library growth

Five nested libraries, the same 2,000 queries.

Single-photograph construction from the full image

Table 1 of the paper.

‡ adapted path · § timed in the method's own API run

What each component is worth

Correctly built assets lost when one component is removed (points).

Cluttered images

50 COCO photos; the pipeline finds the objects itself.

575objects detected

563of them pass the physical gate

172of 237 annotated objects found (IoU ≥ 0.5)

162of 237 pass; every object passes in 15 of 50 images

Side by side with every external method

The same inputs through every method, in the same simulator.

Citation

BibTeX

@misc{yang2026affordcraft,
  title        = {AffordCraft: Scalable Construction of Task-Ready Simulation Assets from Single Images},
  author       = {Yang, Haoyun and Zhou, Xueyang and Xie, Ziyi and Chen, Yongchao},
  year         = {2026},
  howpublished = {\url{https://affordcraft.github.io/}}
}

Library assets come from PartNet-Mobility, Objaverse, Google Scanned Objects and YCB; photos from Open Images and COCO.

AffordCraft

From one photograph to a robot that opens the door.

From a photo to a scene a robot can work in.