Image-to-video rather than text-to-video

Two tests, one verdict

Roger, our house actor, was built once: seven views under flat light, a description block repeated word for word. Then we wanted to animate him. First test: one sentence, “Roger takes his glasses out of his jacket”, sent to a video generator with the character as a reference. The result moved well. It was not Roger: another face, another haircut, the jacket more or less. Second test: the same sentence, but starting from an already approved picture from the casting sheet, set as the first frame of the shot. Roger stayed Roger, from the first frame to the last, hand to the lower back, then the glasses.

Both tests can be seen on his page. Since then, the rule holds for all our characters.

What it changes

Text-to-video asks the model to invent an image and then make it move. Two inventions at once, and the first one drifts. Image-to-video separates the two: we first settle a still image, fix it until it is right, and only then ask it for motion. The video model starts from what it sees; if it sees the right person, it keeps her.

Concretely, every shot in our films has one more step: the start frame. That is where we settle everything animation cannot fix: the frame, the costume, the period, the direction of a gaze, the place of an object in the hand, the absence of a face to camera when we do not want one. It is longer at the start. It is much shorter at the end, because we no longer regenerate a shot ten times for an eyebrow.

Where text-to-video keeps its place

For a set with no character, a mood, a cutaway, text-to-video helps: it invents fast and the drift does not show. As soon as a recurring character, a signature object or a period detail enters the frame, we go back through the image.

What we keep from it

A generated character does not exist in the model; it exists in its reference images. As long as we start from them, it holds. The moment we describe it, we recast it.