How we compare AI video models for filmmaking
The rubric Sundream uses to judge video models for narrative work — consistency, direction-following, and cost per usable shot.
Sundream ·
Most video-model comparisons rank single clips: prompt goes in, prettiest eight seconds wins. That's the wrong test for filmmaking. A film is thirty to a few hundred shots that have to agree with each other — same faces, same light, same lens language — and the model that wins on a one-off clip is often not the model that survives a twelve-scene production.
This post explains the rubric we use inside Sundream when we evaluate a model for narrative work. In follow-ups, we'll publish measured comparisons against this rubric as we run them.
What we measure
Character consistency
Can the model hold a face across shots when given the same character reference? We test the same character in different framings (wide, tracking, close-up) and different lighting, and judge whether an audience would accept it as one person.
Direction-following
Filmmaking prompts are camera direction, not vibes: "slow push-in," "track right as she crosses frame," "insert on the receiver." We test whether the model executes the instruction or ignores it in favor of what it prefers to render.
Temporal coherence within a shot
Warping hands, morphing props, and objects that pop in and out cost a shot its believability. We look at how long a shot can run before coherence breaks down.
Style stability across a sequence
Given a style reference, does shot 14 look like it belongs to the same film as shot 2? Models drift in different ways — some lose the palette, some lose the grain, some lose the era.
Cost per usable shot
Raw per-clip price is misleading. What matters is the price of a shot you keep: a cheap model that needs six attempts per usable shot is more expensive than a pricier one that lands in two. We track attempts-to-keep across our test scenes and fold it into the effective cost.
Why this lives in a studio, not a leaderboard
Model rankings shift with every release, and the right pick often differs shot by shot — the model that nails a dialogue close-up may lose a wide landscape. That's why Sundream lets you choose the model per shot while the project carries your references and story context to whichever one you pick. When a better model ships, your film doesn't start over.
We'll publish rubric results for specific model matchups here as we complete them. If there's a comparison you want to see, tell us.