How we compare AI video models for filmmaking

Most video-model comparisons rank single clips. A prompt goes in, the prettiest eight seconds wins. That does not answer all the questions needed for filmmaking. A film is thirty to a few hundred shots that have to agree with each other on faces, light, and lens language, so a strong one-off clip does not establish reliability across a production.

A practical starting point

Compare a dialogue close-up, a moving wide shot, and a prop interaction using the same cast and visual references. Repeat each setup, define what would make a clip usable before judging it, and record the cost of every attempt. Choose based on the clips you could actually edit together.

This post proposes a rubric, not a measured video-model ranking. Our two-character dance editing case study now documents one practical video comparison, including rejected attempts. It is not a repeated benchmark against every criterion below.

Can it hold a face?

Given the same character reference, does the model keep the same person across shots? A useful test places one character in different framings (wide, tracking, close-up) and different lighting, and asks whether an audience would accept every shot as the same actor. Score each framing separately; success on a close-up does not tell us how the same reference will hold in a wide shot.

Does it take direction?

Filmmaking prompts are camera direction, not vibes: slow push-in, track right as she crosses frame, insert on the receiver. Check whether the model executes the instruction or quietly substitutes whatever it prefers to render. A model that ignores the shot list is a model you have to fight for every setup.

How long before it comes apart?

Warping hands, morphing props, and objects that pop in and out cost a shot its believability. Record how many seconds a shot can run before coherence breaks, because that number sets the longest shot you can plan.

Does shot 14 look like shot 2?

Given a style reference, does a shot late in the sequence still belong to the same film as one early in it? Models drift in different ways. Check for changes in palette, grain, and era. Any of them breaks the film.

What does a shot you keep actually cost?

Per-clip price is only part of the cost. What matters is the price of a shot that makes the cut. For example, six $0.20 attempts cost $1.20 for one usable shot; two $0.50 attempts cost $1.00. Those are illustrative prices, not measured model rates. Divide the total billed generation cost, including rejected attempts, by the number of usable shots. If none are usable, report that explicitly rather than a misleading cost of zero. Report clip length and settings alongside cost so the comparison is meaningful.

Why this lives in a studio, not a leaderboard

Rankings shift with every release, and the right pick often differs shot by shot. The model that nails a dialogue close-up may lose the wide landscape. So Sundream lets you choose the model per shot, and the project carries your references and story context to whichever one you pick. When a better model ships, your film does not start over.

What we have measured so far

Our published writing and image studies use repeated generations and record cost alongside quality. Their judging protocols differ: the writing and style studies describe blinded evaluation; the storyboard study also documents visual corrections to its automated judge. None is a video-model benchmark:

• Making AI storyboard sheets reliable, five rounds with Seedream 4.5 and the panel-detection fix for vertical formats.
• Words, a reference image, or an AI description?, 252 blinded tests of style conditioning across five visual styles.
• We had 8 LLMs write 104 screenplay scenes, a pairwise comparison of writing models under five directorial references.

Our first video character-replacement case study compares Wan, MiniMax and Kling outputs and records Omni and Aleph failures. Repeated video-model matchups against the full rubric remain future work. If there is a comparison you want to see, tell us.