Abstract

In this paper, we address the challenge of generating temporally consistent videos with motion guidance. While many existing methods depend on additional control modules or inference-time fine-tuning, recent studies suggest that effective motion guidance is achievable without altering the model architecture or requiring extra training. Such approaches offer promising compatibility with various video generation foundation models. However, existing training-free methods often struggle to maintain consistent temporal coherence across frames or to follow guided motion accurately. In this work, we propose a simple yet effective solution that combines an initial-noise-based approach with a novel motion consistency loss, the latter being our key innovation. Specifically, we capture the inter-frame feature correlation patterns of intermediate features from a video diffusion model to represent the motion pattern of the reference video. We then design a motion consistency loss to maintain similar feature correlation patterns in the generated video, using the gradient of this loss in the latent space to guide the generation process for precise motion control. This approach improves temporal consistency across various motion control tasks while preserving the benefits of a training-free setup. Extensive experiments show that our method sets a new standard for efficient, temporally coherent video generation.

Overview of our method. We first conduct (a) inversion noise initialization on the reference video to obtain the initial noise z_T (Section 3.2). Then we (b) extract the motion pattern M from the reference video for each tracked point p (Section 3.3). During the (c) denoising process, we use the proposed frame-to-frame motion consistency loss L_c , calculated with Eq. (4) based on M and newly extracted M^' from the noise z_t as the motion guidance for the noise estimation (Section 3.4).

Our method can better follow the trajectory movement and maintain the content consistency across different frames.

We use iPhone 15 pro max to take hand moving. Our method can successfully simulate the motion of fingers.