How we test and score

Generation

• Fixed prompts: 20 public prompts (12 video, 8 image). See the prompt library.
• Identical settings: October 2026 video clips were 4 seconds at 720p, 16:9 (prompt 9 at 9:16), with each model's default options. Veo 3.1 Fast and Kling 3.0 had audio on. Gen-4.5 doesn't generate audio. Images: Nano Banana 2.1 at 1024², Ideogram 4.0 at 2048², medium quality.
• Access: everything ran through the Runway Dev API. Results from each tool's own app can differ.
• No cherry-picking: one generation per model per prompt. A retry is used only if a generation fails outright. We publish every output we got.

Scoring

• Rubric (1–10 each): Prompt adherence (did it do what was asked?), Motion / physics (video only: is the movement plausible and coherent?), Visual quality (lighting, detail, aesthetics), Artifacts (hands, faces, text, morphing; higher = cleaner).
• Overall is the plain average of the criteria. A model's monthly score is the average across its prompts.
• Without model names: for each prompt, outputs were shuffled into lettered contact sheets (4 frames per video clip) and scored before the model names were revealed. Audio and lip-sync weren't scored, because we judged from frames.
• Every score is shown with its reasoning on the all-outputs page.
• No pay-to-win: sponsors and affiliate partners never see or influence scores.

Limits

• One generation per prompt is a small sample, and models are random. Treat the rankings as a well-documented snapshot, not a final ruling.
• Prompt 10 (image-to-video) wasn't run in October.

Corrections: [email protected]