Preprint

Beyond the Visible

Learning Dense 4D Motion in Contact-Rich Deformable Worlds

A vision-conditioned Eulerian displacement field that predicts dense 4D motion everywhere: visible surface, occluded interior, and ambient off-object space.

Li-Xuan Peng1,†, Po Yi Wu2,*, Yu-Tsen Tsai2,*, Tsung-Wei Ke2,‡, Min Sun1,‡
1EE, National Tsing Hua University  ·  2CSIE, National Taiwan University
†Corresponding author  ·  *Equal contribution  ·  ‡Equal advising
Observed RGB-D point cloud: only the visible surface is populated.
DELPHI's predicted dense field: motion filled in through the occluded interior and ambient space.
Visible  RGB-D Beyond the visible
↔
visible occluded interior ground-truth
Drag: left is what the scene-flow-based method predict, right is the field DELPHI predicts through the occluded interior.

Overview

Decouple the dynamics from the view

We propose DELPHI, a model, and ADYTON, a benchmark, for predicting 4D deformable-object interaction dynamics from a single monocular RGB-D video. Deformable dynamics are non-Markovian in observation space: a frame exposes a front-facing surface, but occluded back faces and interiors carry the state that governs the next instant — momentum in flight, stored elastic energy, plastic history.

Tracking observed points is a Lagrangian commitment: the representation is defined only on the entities you already see, so the state you need leaves it the moment a face rotates out of view or a contact buries it.

To learn complete dynamics regardless of observation completeness, decouple the particle representation from the vision-conditioned dynamics.

Concretely, DELPHI is an operator: from a visual context \(\mathcal{C}\) it produces a whole Eulerian displacement field over all of \(\mathbb{R}^3\), which is then queried at any coordinate — visible surface, occluded interior, or ambient off-object space — without re-tracking anything.

\[ \mathcal{G}_\theta:\; \mathcal{C}\;\longrightarrow\;\bigl(\mathbb{R}^3\rightarrow\mathbb{R}^3\bigr), \qquad u(x)=\mathcal{G}_\theta(\mathcal{C})(x)\;\;\text{for any}\;\; x\in\mathbb{R}^3. \]

Main Results · Full Geometry

Single-step full-geometry prediction

One prediction from one observation context, over the full geometry. Rotate the 3D scene, scrub the timeline, and toggle entity paths — visible surface, occluded surface, interior, ambient — against ground-truth material point displacements.

Interactive 3D viewer
Loads when you scroll here — the Rerun runtime is a one-time ≈12 MB download,
cached afterwards. Rotate, scrub, and switch scenarios with the tabs above.
visible surface occluded surface interior field ambient off-surface ground-truth material points (MPM)

The Benchmark · ADYTON

Ground truth where you cannot look

ADYTON is a Genesis-MPM simulation benchmark of 12,200 dynamic scenes, each a monocular RGB-D video from a static camera paired with dense material-point displacements sampled uniformly through the entire object volume — visible surface, occluded surface, and interior alike. It spans 18 objects across 9 categories, elastic and elastoplastic, driven through 10 motion patterns in three families. Camera, lighting, floor texture, and initial motion are randomized every scene, and the held-out split probes OOD generalization to novel object combinations, interactions, and layouts.

12,200
dynamic scenes · 10k train / 2.2k test
18
objects · 9 categories
10
motion patterns · 3 families
100%
volumetric point labels
FamilyPatternMaterial
Free / Gravityfree fallElastic
projectileElastic
spin dominantElastic
Collisiontwo-object collisionElastic
glancingElastic
multi-object collisionElastic
Deformationtemporal deformation (collision)ElastoPlastic
temporal deformation (fall)ElastoPlastic
persistent deformation (collision)ElastoPlastic
persistent deformation (fall)ElastoPlastic

Elastic: 5 geometries × 2 stiffness/density variants (10 objects). Elastoplastic: 4 geometries × 2 yield-stress variants (8 objects). Deformation patterns reuse the Free/Gravity and Collision motion profiles.

Glimpse the raw data

Raw ADYTON scenes with their saved Rerun blueprints, before any prediction. Rotate and scrub the ground-truth material-point motion the model is trained against.

ADYTON data viewer
Loads when you scroll here — the Rerun runtime is a one-time ≈12 MB download,
cached afterwards. Rotate, scrub, and switch scenes with the tabs above.

Additional experiment · zero-shot rollout

Autoregressive rollout at inference time

Feeding the field its own predictions rolls the dynamics forward with no retraining — a zero-shot, inference-time capability. Rotate, scrub, and toggle entity paths; switch camera setup with the toggle above.

Interactive 3D viewer
Loads when you scroll here — the Rerun runtime is a one-time ≈12 MB download,
cached afterwards. Rotate, scrub, and switch scenarios with the tabs above.
observed surface points predicted interior field ground-truth material points (MPM)

Method · DELPHI

How the field is built

DELPHI reads a monocular RGB-D video and outputs an implicit displacement field over the entire 3D volume, covering visible surfaces, occluded surfaces and object interiors. The field is parameterised by the video's visual tokens, and any 3D point retrieves its motion by cross-attending to them. All positions and motions are expressed in the current camera's frame.

DELPHI architecture: an Any4D backbone encodes the input RGB-D frames into multi-scale spatiotemporal visual tokens; a 3D query point cross-attends to them with a relative spatial bias, and an MLP decodes the displacement at that point.
DELPHI architecture. Multi-scale visual tokens from the Any4D backbone (top) parameterise the field; a query point encoded by an MLP cross-attends to them with a relative spatial bias, and an MLP decodes its displacement (bottom).
01 · tokens

Video as visual tokens

DELPHI adopts Any4D pretrained on diverse 4D reconstruction tasks, extracts per-frame features that are concatenated over the history. These tokens carry geometric, motion and temporal cues, and they parameterise the output field.

02 · query

Pointwise queries

A lightweight MLP encodes any 3D position \(x\) into a query token. Each point is queried independently, which decouples inputs from outputs and makes the field inference highly parallel.

03 · field

Cross-attention decoding

The query cross-attends to all visual tokens, and an MLP decodes the result into a displacement: \(\mathcal{G}_\theta(\mathbf{o}_k)(x)=\mathrm{MLP}(\mathrm{CrossAttn}(q,F_k))\).

Relative spatial biasEach token sits at the centroid of its back-projected patch. The offset \(x-p^i_s\) passes through Fourier features \(\gamma\) and an MLP \(\phi\) and is added to the attention logit, \(\operatorname{softmax}\big(q^{\top}f^i_s/\sqrt{d}+\phi(\gamma(x-p^i_s))\big)\). Because the bias sees both distance and direction, it learns motion-specific priors, such as looking below a falling object or ahead of a sliding one.
Multi-scale tokensA purely local bias makes the model nearsighted about the far future. Average-pooling fine tokens into coarse ones, each located at the centroid of the union of its pixels, gives fine-to-coarse context that the query attends to with the same spatial bias.
LoRA adaptationAny4D was pretrained on real-world data, so LoRA adapts the backbone to our synthetic scenes. DELPHI is trained with an L2 loss on displacements.

Nomenclature: Drawn from the sacred topography where Zeus's eagles marked the center of the earth to divine unseen futures, Delphi was revered as the navel of the world and the seat of prophetic counsel. It mirrors our ambition towards the World Model, acting as an anywhere-queryable continuum that answers queries at any coordinate in space. Adyton, the oracle's inviolate inner sanctuary, embodies our benchmark, holding the elusive physical truths that nature withholds yet our model strives to decipher.

Acknowledgement

Compute Support

This work was supported by the AMD University Program AI & HPC Cluster. We further acknowledge Kuo-Guang Tsai for his technical support on the AMD system cluster infrastructure.

We also gratefully acknowledge the support of NVIDIA Corporation through the NVIDIA Academic Grant Program (2025), which provided the computing hardware used in this research.

Citation

BibTeX

@inproceedings{peng2026delphi,
  title     = {Beyond the Visible: Learning Dense 4D Motion in
               Contact-Rich Deformable Worlds},
  author    = {Peng, Li-Xuan and Wu, Po Yi and Tsai, Yu-Tsen and Ke, Tsung-Wei and Sun, Min},
  booktitle = {arXiv},
  year      = {2026}
}