Co-Generating 3D Motion and Video for Any Rig
| Source: HF Papers | Original article
A new approach co‑generates 3D motion and video for arbitrary rigs, letting video models animate specific 3D assets and replay the motion within their scenes.
Nvidia researchers have unveiled **Joints, Camera, Action! (JCA)**, a new model that simultaneously generates a video clip and a synchronized 3‑dimensional motion trajectory for any user‑supplied rig, regardless of its topology. By linking a pretrained video generator with a dedicated motion branch through cross‑attention, JCA produces not only realistic pixel‑level footage but also a replayable 3‑D animation and a camera pose that registers the motion within the scene. The system can handle arbitrary skeletons – from a rigged monkey clearing a high‑jump bar to custom key‑frame‑driven sequences – and keeps the motion component separate from the visual generator, sidestepping the massive parameter counts of prior human‑only approaches.
The breakthrough matters because today’s video‑generation models excel at visual realism yet leave the underlying motion trapped in pixels, limiting their utility for game developers, VFX studios, and robotics engineers who need reusable animation data. JCA’s ability to output a full motion trajectory that can be applied to a specific 3‑D asset opens a direct pipeline from generative AI to production‑ready animation, potentially cutting the time and cost of manual rigging and motion‑capture.
Watch for early integrations of JCA into existing content‑creation suites and for open‑source releases of the code and model weights. Follow‑up work may explore real‑time control, higher‑fidelity camera tracking, and extensions to multi‑modal editing, building on the same cross‑attention framework that bridges video and motion generation.
Sources
Back to AIPULSEN