PhyEdit: Towards Real-World Object Manipulation via Physically-Grounded Image Editing

Ruihang Xu

Dewei Zhou

Xiaolong Shen

Fan Ma ^*

Yi Yang

ReLER Lab, CCAI, Zhejiang University

^* Corresponding author

ACM MM 2026 Poster

Paper arXiv Code Model Dataset

Comparison of PhyEdit editing results against Nano Banana Pro across three physical scenarios — Comparison of image editing results on three physical scenarios between PhyEdit and Nano Banana Pro. PhyEdit achieves more accurate object placement and geometric consistency.

PhyEdit enables physically grounded image editing by manipulating objects in 3D with geometric accuracy and physical consistency.

Abstract

Achieving physically accurate object manipulation in image editing is essential for its potential applications in interactive world models. However, existing visual generative models often fail at precise spatial manipulation, resulting in incorrect scaling and positioning of objects. This limitation primarily stems from the lack of explicit mechanisms to incorporate 3D geometry and perspective projection. To achieve accurate manipulation, we develop PhyEdit, an image editing framework that leverages explicit geometric simulation as contextual 3D-aware visual guidance. By combining this plug-and-play 3D prior with joint 2D–3D supervision, our method effectively improves physical accuracy and manipulation consistency. To support this method and evaluate performance, we present a real-world dataset, RealManip-40K, for 3D-aware object manipulation featuring paired images and depth annotations. We also propose ManipEval, a benchmark with multi-dimensional metrics to evaluate 3D spatial control and geometric consistency. Extensive experiments show that our approach outperforms existing methods, including strong closed-source models, in both 3D geometric accuracy and manipulation consistency.

Demo

Qualitative comparisons on ManipEval against strong closed-source models Qwen-Image-2.0-Pro and Nano Banana Pro. Correct and incorrect manipulations are marked in green and red dashed boxes, respectively.

Continuous Object Manipulation

Beyond single-step editing, PhyEdit follows a user-defined 3D trajectory while preserving geometric consistency. The out-of-distribution robotic arm highlights the model’s generalization.

Initial scene and user-defined 3D manipulation trajectory — Initial scene and user-defined 3D trajectory.

Solid boxes mark keyframes generated by PhyEdit; dashed boxes mark interpolated frames.

Interactive GUI

The interactive interface supports object segmentation, direct 3D manipulation with live geometric previews, and final image generation with PhyEdit.

Workflow from object selection and 3D manipulation to the final edit.

Method

PhyEdit augments a DiT image editor with explicit 3D guidance. A 3D transformation module constructs a depth-aware preview for the backbone, while joint latent- and depth-space losses enforce appearance and geometric consistency.

Overview of the PhyEdit model architecture — Overview of PhyEdit. The 3D transformation module converts user controls into a geometric preview for the DiT backbone, trained with joint latent and depth supervision.

3D Transformation Module

Given a source image $I_\text{src}$ , object mask $M_o$ , and translation $\Delta\mathbf{p}_o$ , we estimate depth $D$ and camera pose $(R, t)$ , transform the masked object in 3D, and reproject it:

\mathbf{P}_{o}=\operatorname{Unproj}(I_{\text{src}}, M_{o},D,R,t), \quad \mathbf{P}_{o}'=\mathbf{P}_{o}+\Delta\mathbf{p}_o, \quad I_{\text{prev}}=\operatorname{Proj}(\mathbf{P}'_o,R,t)

$I_\text{prev}$ provides explicit geometry to the DiT backbone and supports simultaneous multi-object manipulation.

Joint Supervision

We complement the flow-matching loss with scale-invariant logarithmic (SILog) depth supervision:

\mathcal{L} = \mathcal{L}_{\text{flow}} + \lambda_d \mathcal{L}_{\text{depth}}

where $\lambda_d = 0.1$ balances the two terms. This auxiliary supervision requires minimal changes to DiT-based editors.

RealManip-40K Dataset

RealManip-40K contains 41,154 real-world source-target pairs with 3D object motion, including depth-axis changes. Each pair includes depth maps, object masks, and representative 3D coordinates.

The four-stage pipeline used to construct RealManip-40K — Dataset construction pipeline: data source filtering → camera-static clip extraction → depth and mask processing → depth-aware frame pair selection.

DBSCAN-based camera token clustering selects near-static clips before depth, mask, and frame-pair processing.

Experiments

Quantitative Results on ManipEval

Method	DIoU ↑	Mask IoU ↑	AbsRel ↓	δ₁.₂₅ ↑	Chamfer ↓	Centroid ↓	RA-DINO ↑	DeQA ↑	Phys-VLM ↑
Qwen-Image-Edit	53.28	13.80	64.63	40.61	46.31	52.48	26.09	75.67	90.55
LightningDrag	53.81	18.90	57.07	35.99	45.76	53.57	21.92	73.60	88.60
ChronoEdit	48.92	9.26	65.80	36.35	45.02	54.40	21.73	75.29	92.15
GPT-Image-1.5†	52.33	11.34	70.95	36.50	39.79	52.78	23.80	75.82	88.68
Qwen-Image-2.0-Pro†	56.48	15.60	55.93	41.39	29.64	43.48	31.04	68.26	82.91
Nano Banana Pro†	59.97	18.93	55.02	46.11	25.33	35.62	34.77	77.48	91.06
Ours	65.33	27.20	49.53	51.08	18.93	32.12	36.91	75.48	93.72

† Proprietary commercial model. All metrics normalized to [0, 100].

Our method achieves the best overall performance across manipulation-related metrics, outperforming strong closed-source commercial systems on geometry-sensitive measures. Against Nano Banana Pro: DIoU +5.36, Chamfer distance −6.40, RA-DINO +2.14. Our lead over commercial baselines grows further on multi-object scenes (Chamfer gap: 5.19 → 6.58; δ₁.₂₅ gap: 1.91 → 8.02).

BibTeX

@misc{xu2026phyeditrealworldobjectmanipulation,
  title={PhyEdit: Towards Real-World Object Manipulation via Physically-Grounded Image Editing},
  author={Ruihang Xu and Dewei Zhou and Xiaolong Shen and Fan Ma and Yi Yang},
  year={2026},
  eprint={2604.07230},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2604.07230},
}