VIRES: Video Instance Repainting with Sketch and Text Guidance (2024)

2.1 Diffusion-based image generation

Diffusion models[34, 6] have demonstrated remarkable advancements in image generation[31, 33]. These models learn to reverse the forward diffusion process, generating images by iterative denoising a sample from a standard Gaussian distribution. To control this generation process, researchers use adapter mechanisms to incorporate additional conditions[50, 22].Leveraging the generative priors encapsulated in the pre-trained generation model, users can customize content via fine-tuning[32, 17] and edit arbitrary images via training-free strategies[21, 24].These advancements expand the application of diffusion models to various image-based tasks, e.g., super resolution[39], image colorization[41], and reflection removal[8].Given the similar spatial representations between images and videos, researchers are inspired to effectively lift these pre-trained image priors to the video domain.

2.2 Diffusion-based video generation

Early text-to-video generation works attempt to extend the UNet-based denoising network with space-only 3D convolutions, training them from scratch[7].However, this approach proves computationally expensive, leading researchers[36, 5] to explore fine-tuning pre-trained text-to-image models[31]. This fine-tuning typically introduces additional temporal modules (e.g., temporal convolutions[35] or temporal attention[2]) to improve the temporal consistency, while freezing most of the pre-trained denoising network weights to preserve spatial generative priors.To further reduce generation costs, researchers explore zero-shot text-guided video editing that solely performs during the inference phase[14, 51].Recently, impressive text-to-video generation results from OpenAI[20] have renewed interest in training models from scratch, but this time leveraging the diffusion transformer architecture[52, 53, 48].Given the significant improvements demonstrated on video generation benchmarks[10], we build upon this foundation and develop modules for controllable video generation.

4.1 Overview

Given an $N$-frame original video clip $x=\{x^{i}\}_{i=1}^{N}$, a corresponding sketch sequence $s=\{s^{i}\}_{i=1}^{N}$ presenting the structure, a text description $y$ describing the appearance, and an instance mask sequence $m=\{m^{i}\}_{i=1}^{N}$ indicating the repainted regions, VIRES is proposed to repaint the specific video instance. The overall pipeline is illustrated inFig.2.

Input encoding. The original video clip $x$ is encoded into the latent code $z$ using the pre-trained spatial-temporal VAE encoder $\mathcal{E}^{\mathrm{x}}$[52] as $z=\mathcal{E}^{\mathrm{x}}(x)$.The sketch sequence $s$ is processed by our proposed Sequential ControlNet $\mathcal{F}^{\mathrm{s}}$ to effectively extract structure layouts as $f^{\mathrm{s}}=\mathcal{F}^{\mathrm{s}}(s)$.The text $y$ is encoded with the pre-trained text encoder $\mathcal{F}^{\mathrm{y}}$[29] to obtain the word embeddings as $f^{\mathrm{y}}=\mathcal{F}^{\mathrm{y}}(y)$.

Condition injection. To adaptively capture the high-contrast structure details of the sketch sequence, we introduce the standardized self-scaling.We further augment the DiT backbone with the sketch attention, which interprets and injects fine-grained sketch semantics into the latent space.Word embeddings are injected into the DiT backbone with the pre-trained cross-attention modules.

4.2 Sequential sketch feature extraction

As presented inFig.2 (c), we propose the Sequential ControlNet to extract structure layouts from the sketch sequence. This is followed by the standardized self-scaling, designed to adaptively capture the high-contrast structure details of the extracted features.

Sequential ControlNet.Previous ControlNet[50] is designed for image editing, which lacks temporal consistency and can produce flickering artifacts when applied to video-based tasks. This motivates us to introduce Sequential ControlNet^††Further variations are discussed in the Supp. for the video instance repainting.Our Sequential ControlNet includes convolutional layers, residual blocks, and downsampling layers. Each convolutional layer consists of a 3D causal convolution[49], followed by a Group Normalization[43] and a SiLU activation function[3], to effectively capture spatial-temporal dependencies between frames.Residual blocks consist of two such convolutional layers with a residual connection. Downsampling layers are convolutional layers with a stride of 2, applied to compress the spatial or spatial-temporal dimensions.Since the spatial-temporal VAE encoder[52] downsamples the original video clip by $4\times$ temporally and $64\times$ spatially, we adopt one spatial downsampling layer and two spatial-temporal downsampling layers to match the feature map dimensions of the DiT backbone.To align the embedding channels, we progressively increase the number of channels. Specifically, we increase the embedding channels in the shallow convolutional layers and double the channels at each downsampling layer.The final three layers maintain a high channel dimension to effectively extract structure layouts from the sketch sequence.

4.3 Latent sketch semantic interpretation

As illustrated inFig.2 (d), following the standardized self-scaling that provides the high-contrast structure details, we further augment the DiT backbone with the sketch attention to interpret and inject fine-grained sketch semantics within the latent space.

DiT backbone.The VIRES framework builds upon the pre-trained text-to-video generation model[52], leveraging its DiT architecture that consists of stacked transformer blocks. Each block incorporates separate spatial and temporal self-attention modules to capture intra-frame and inter-frame dependencies, respectively. In each module, Self-Attention (SA) and Feed-Forward Network (FFN) extract contextual features, Cross-Attention (CA) injects semantics from word embeddings, and the scale and shift (S&S) and the gate modulate the latent code with timestep embeddings.We concatenate the latent code with the instance mask before feeding it to the transformer, and patchify both the latent code and the sketch sequence into non-overlapping $1\times 2\times 2$ tokens, resulting in a $4\times$ spatial downsampling.

4.4 Visual sketch texture guidance

As illustrated inFig.2 (e), during decoding the latent code into a video clip, we additionally adopt a sketch-aware encoder, which provides multi-level texture features to the spatial-temporal VAE decoder and aligns the structure of repainted results with the provided sketch sequence.

Spatial-temporal VAE.We use the pre-trained spatial-temporal VAE[52] to encode the original video clip into a latent code and subsequently decode it back into a video clip.Specifically, the spatial-temporal encoder firstly uses a stack of 2D vanilla convolution blocks with three downsampling layers, achieving $64\times$ spatially compression. Following this, 3D causal convolution blocks capture spatial-temporal dependencies with two downsampling layers applied only to the temporal dimension, resulting in $4\times$ temporal compression.The spatial-temporal VAE decoder mirrors the architecture of the encoder, using 3D causal convolutions for spatial-temporal modeling and upsampling the temporal dimension before the spatial dimension.

5.1 Quantitative evaluation metrics

We quantitatively evaluate performance using five metrics:(i) the Peak Signal-to-Noise Ratio (PSNR)[11] for the visual perceptual quality;(ii)the Structural Similarity Index Measure (SSIM)[40] for spatial structure consistency;(iii)the Warp Error (WE), calculated by warping repainted video clips with the estimated optical flow, following VidToMe[18] to evaluate motion accuracy;(iv) the Frame Consistency (FC), measuring cosine similarity between consecutive frames for temporal consistency using the CLIP image encoder[28];and (v) the Text Consistency (TC), measuring cosine similarity between repainted video clips and text descriptions for text-video alignment using the VideoCLIP-XL[37].

5.2 Comparison with state-of-the-art methods

We compare our approach with state-of-the-art video editing methods for generating 51-frame video clips, including VideoComposer[38], Text2Video-Zero[14], Rerender[47], VidToMe[18], and RAVE[13] using their publicly released models with default configurations.

Qualitative comparisons.We present visual quality comparison results with aforementioned methods [38, 14, 47, 18, 13]. As shown inFig.3, the left column edits instance appearance using text descriptions while preserving structure with extracted sketches. Conversely, the right column maintains its appearance using annotated text descriptions and edits structure via sketch modifications.As a result, Rerender [47] produces excessively smooth textures (e.g., Fig.3 left, the unnatural smoothness of the horse and tree).VidToMe[18] generates a cartoonish effect (e.g., Fig.3 left, the stylized appearance of the entire frame).Text2Video-Zero[14] presents inaccurate color tone (e.g., Fig.3 right, the white jersey of the player).RAVE[13] suffers from inter-frame flickering (e.g., Fig.3 right, the player’s neck flashing with frame brightness fluctuations).VideoComposer[38] struggles to follow the target structure (e.g., Fig.3 right column, the missing sleeves of the player).In contrast, our VIRES faithfully repaint the video instance with the sketch and text guidance.

Quantitative comparisons.We show comprehensive quantitative comparison results inTab.1, demonstrating that VIRES outperforms relevant state-of-the-art methods across all five quantitative metrics on both datasets.Specifically, VIRES achieves the best results in visual perceptual quality (PSNR), spatial structure consistency (SSIM), frame motion accuracy (WE), consecutive frame consistency (FC), and text description consistency (TC).

User study.In addition to qualitative and quantitative comparisons, we conduct two user studies to evaluate human preference for our results:(i) Visual Quality Evaluation (VQE):Participants are shown the edited results produced by VIRES and relevant video editing methods. They are asked to select the most visually pleasing video clip.(ii) Text Alignment Evaluation (TAE):Given a corresponding text description, participants are instructed to select the video clip from the same set of edited results that best matches the description.For each experiment, we randomly select 10 samples from each dataset, and recruit 25 volunteers from Amazon Mechanical Turk (AMT) to provide independent evaluations. As shown in Tab.1, our model achieves the highest preference scores in both experiments.

5.3 Ablation study

We discard several modules and establish four baselines to study the impact of the corresponding modules. The evaluation scores and repainted results of the ablation study are presented in Tab.1 and Fig.4, respectively.

W/o Standardized Self-Scaling (SSS).We replace the standardization self-scaling with simple addition when initially injecting the structure layout. This leads to distorted textures in the repainted results (Fig.4 third column, an unusual texture appears around the fish’s eye).

W/o Self-Scaling (SS).We discard the self-scaling, but preserve the standardization for the extracted structure layout before adding them to the DiT backbone. As a result, the model struggles to capture structure details (Fig.4 second column, the fish head appears less detailed).

W/o Sketch Attention (SA).We remove the sketch attention in the DiT backbone, thereby removing the third stage of training. This prevents the model from accurately interpreting fine-grained sketch semantics (Fig.4 fourth column, abrupt yellow patches appear on the fish’s body).

W/o Sketch-aware Encoder (SE).We disable the sketch-aware encoder that provides multi-level features to guide the decoding process, resulting in repainted results that are not well aligned with the expected visual representations (Fig.4 first column, the fish body appearing blurry and flat).

5.4 Application

In addition to four application scenarios illustrated inFig.1 (i.e., video instance repainting, video instance replacement, custom instance generation, and specified instance removal), we present additional three applications.

Sketch-to-video generation.VIRES is capable of generating entire video clips from sketch sequences with text guidance, not limited to repainting specific instances.This is achieved by discarding the original video content and expanding the instance mask to the entire frame, enabling the training-free sketch-to-video generation (e.g., Fig.5 first row, generating a realistic lotus flower from a provided sketch sequence).

Sparse sketch guidance.To reduce user effort when repainting video instances, we incorporate sparse sketch guidance into the VIRES. Instead of requiring a full sketch sequence, we fine-tune the VIRES using two strategies: (i) random dropout of sketch frames with 20% probability; and splitting the video into $2^{d}$ intervals with 80% probability, where $d\in\{0,\dots,4\}$ is a randomly selected interval index. For each interval, only the first sketch frame is provided, with missing frames replaced by black images. As a result, VIRES can repaint video instances with even a single sketch frame (e.g., Fig.5 second row, repainting the pattern on the person’s clothes with the first sketch frame).

Long-duration video repainting.Due to the computational resource limitation, processing the full long-duration video at once is challenging.However, VIRES can effectively repaint long-duration videos by randomly unmasking the frames during training (e.g., the first $k$ frames).Specifically, we initially repaint the first 51-frame video clip, leveraging the last 17 frames of this clip as a hint to iteratively repaint the subsequent 34-frame clips until the entire video is processed. We present the results in the Supp.

7.1 Variations of Sequential ControlNet

We present four typical architectures of Sequential ControlNet inTab.2, where “Conv” denotes a convolutional layer, "Block" means a residual block, and "Down" is a downsampling layer. Numbers in brackets indicate the input and output channel dimensions, respectively. To determine the optimal architecture under constrained computational resources, we train these variations for 10K steps, excluding the sketch attention and sketch-aware encoder. Quantitative results on VireSet are shown in Tab.3, and the best-performing architecture is selected for our model.

7.2 Validation of additional conditions

VIRES is designed specifically for sketch-based video instance repainting, adopting the Standardized Self-Scaling (SSS) to extract condition features. To explore its generalization to other conditioning signals, we augment our dataset with edge maps[4] and depth maps[46] to provide the structure guidance. We then retrain VIRES from scratch on these conditions. Due to limited computational resources, we train for 10K steps with edge maps as a preliminary investigation and 30K steps with depth maps due to its slower convergence. Finally, we compare two VIRES variants: one using SSS and the other using simple addition for feature extraction.As shown inTab.4, SSS provides considerable advantages for edge maps, but only marginal improvements for depth maps. We suggest that this difference arises because both sketch and edge maps have high-contrast transitions between black lines and the white background, allowing self-scaling to effectively capture structure details. In contrast, depth maps are smoother and lack such sharp transitions, limiting the benefits of the self-scaling operation. We further present the qualitative results inFig.6.

7.3 Robustness of sparse conditions

VIRES allows users to provide sparse sketch guidance for video instance repainting, minimizing user effort.To investigate the impact of sketch guidance sparsity on repainting performance, we evaluate VIRES on the VireSet, using varying interval indices $d\in\{0,\dots,4\}$, corresponding to $D=2^{d}$ sketch frames, and a full set of $D=51$. As shown inTab.5, even with a single sketch frame, VIRES can produce high-quality results (PSNR and SSIM) with robust temporal consistency (WE and FC).

7.4 Compatibility with DiT backbone

In this paper, we build VIRES upon the pre-trained OpenSora v1.2[52]. Given that many text-to-video models[53, 48] share a similar DiT backbone architecture, our proposed modules offer potential compatibility with these approaches, including the Sequential ControlNet for layout initialization (Sec. 4.2), the standardized self-scaling for details capture (Sec. 4.2), the sketch attention (Sec. 4.3) for semantic injection, and the sketch-aware encoder for structure alignment (Sec. 4.4).We believe our work will inspire further research on guiding pre-trained text-to-video models and open new avenues for conditional video repainting.

7.5 Organization of supplementary video

We provide a supplementary video to dynamically showcase our repainting results. The video is structured as follows:(i) Typical application scenarios: We demonstrate four typical repainting scenarios and compare our results with relevant methods[47, 18, 14, 13, 38]. Instance repainting/replacement results are shown inFig.7, and instance generation/removal results are inFig.8.(ii) Sketch-to-video generation and inpainting: We demonstrate sketch-to-video generation and conditional video inpainting, comparing our results with VideoComposer[38], as it is the only relevant method that supports this functionality. Results are shown inFig.9.(iii) Sparse sketch guidance: We showcase sparse sketch guidance, repainting two distinct variations of the same video using only two different first sketch frames. This functionality is not supported by existing methods. Results are shown inFig.10.(iv) Long-duration video repainting: We demonstrate repainting on a long-duration (13-second) video, with representative frames presented inFig.11.(v) Comparison and ablation study: Finally, the video includes comparisons with other methods (Sec.5.1) and additional ablation studies (Sec.5.2).To improve visual clarity and facilitate detailed comparison, the video playback speed is halved ($2\times$ slower).