CinemaTraj: Composing Atomic Camera Trajectories for 3D Scenes with LLM Agents

CinemaTraj teaser: from a 3D scene and a natural language prompt, an LLM agent plans a collision-free camera trajectory and renders a narrated cinematic video. — Given a 3D scene and a natural language prompt (*left*), **CinemaTraj** decomposes the request into a sequence of cinematographic movements — orbit, crane, zoom-in — grounded in a 3D scene graph, and plans a collision-free camera trajectory through the scene (*middle*). The resulting trajectory is rendered into a cinematic video with synchronized voiceover and subtitles (*right*).

Abstract

Automatically generating cinematically expressive camera trajectories through 3D scenes from natural language descriptions is a challenging task of high practical value, with applications ranging from real-estate advertising to virtual tour creation. Existing methods either lack true 3D spatial awareness by relying on 2D image priors, or treat trajectory generation as a geometric path planning problem divorced from cinematographic semantics. We present CinemaTraj, a framework that reframes camera trajectory planning as a language-grounded spatial reasoning problem. Given a set of RGB-D images and a user prompt, CinemaTraj equips an LLM agent with a structured 3D scene graph: the agent decomposes the prompt into a sequence of atomic cinematographic movements (dolly, orbit, crane, pan, tilt, zoom, arc). Each movement is instantiated via a novel parametric trajectory representation that is both cinematographically expressive and optimizable for collision avoidance. The scene graph acts as a structured spatial prior, grounding the agent's reasoning in accurate geometric and semantic knowledge of the environment. CinemaTraj further generates synchronized voiceover and subtitles aligned with camera motion, producing narrated cinematic video outputs. We evaluate CinemaTraj on real-world ScanNet++ environments, and show that it produces prompt-faithful, collision-free trajectories with high cinematographic quality, outperforming existing approaches on prompt alignment, trajectory quality, and safety metrics.

Method

CinemaTraj pipeline: Scene Graph Extractor, User Prompt Translator, Parametric Atomic Trajectory Builder, and Collision and Occlusion-Free Trajectory Optimizer. — **Pipeline overview.** (1) The **Scene Graph Extractor** builds a cinematography-oriented scene graph from RGB-D frames, retaining room–object containment and object–structure relations that govern movement feasibility. (2) The **User Prompt Translator** gives an LLM agent the scene graph and a cinematographic toolset, translating the prompt into a sequence of atomic trajectory commands. (3) The **Parametric Atomic Trajectory Builder** selects a score-based anchor viewpoint per object and instantiates each command through an interpretable parametric representation. (4) The **Collision & Occlusion-Free Trajectory Optimizer** refines the free parameters by gradient descent on a signed distance field, avoiding obstacles and occlusions while preserving shot style — an orbit remains an orbit, even in cluttered scenes.

Gallery of atomic camera movements: full/half/quarter orbits, pans, dolly in/out, crane, tilts, zooms, static hold, and the transitional arc. — **Atomic cinematographic movements.** The parametric shot vocabulary: object-centric movements (orbit full/half/quarter, pan, dolly in/out, crane, tilt, optical zoom, static hold) and the transitional *arc* connecting consecutive targets. Frustum color encodes temporal progression (blue → red). The toolset is extensible — new trajectory types can be added without changing the rest of the pipeline.

Open-Ended Prompts

Open-ended prompt results: planned trajectories over four scene reconstructions with numbered keyframes along each tour. — **End-to-end planning from an abstract request.** For the open-ended prompt *“Can you give an incoming Airbnb guest a detailed walkthrough of this house?”*, CinemaTraj autonomously selects salient objects, orders the visits, and composes atomic movements into a full tour. Left: planned trajectories over the scene reconstruction; right: keyframes at the numbered anchors.

Baseline Comparisons

Prompts are color-coded: camera movements and target objects. Each video shows the rendered walkthrough (top) and a bird's-eye visualization of the executed camera trajectory over the scene reconstruction (bottom). Videos play automatically when scrolled into view.

Scene 1 — ScanNet++ `09c1414f1b`

Prompt “I'd like a specific sequence: First, do an orbit around the table, then do a crane shot on the sofa, then zoom in the bicycle, then pan right toward the refrigerator, and finally do a static view of the sink.”

CinemaTraj (Ours)

ChatCam + GenDoP

CCTG

Scene 2 — ScanNet++ `0cf2e9402d`

Prompt “First, do a quarter orbit around the microwave, then move in toward the refrigerator, then pan left from the washing machine, and finally tilt down from the kitchen counter.”

CinemaTraj (Ours)

ChatCam + GenDoP

CCTG

Scene 3 — ScanNet++ `2c7c10379b`

Prompt “Here's what I want: First, move in to the sofa, then do a 360 degrees orbit around the table, then zoom in toward the PC, then do a crane shot of the bed, and finally do a tilt down to the chair.”

CinemaTraj (Ours)

ChatCam + GenDoP

CCTG

Ablation Study

Removing the Anchor Selector leads to incorrect object targeting; removing parametric trajectories yields unstructured paths; removing SDF-based optimization causes geometry collisions.