Vimoo 1.0: Video Generation for Compact, Lightweight, and High-Speed

2.1 Overview

The Vimoo 1.0 family generates videos in a compressed visual latent space [31, 4]. A text encoder converts the prompt into conditioning features, which are projected and refined before entering the conditioning stream. The Wan 2.2 VAE maps images and videos into visual latents and decodes generated latents back into video. For I2V, the reference image supplies both semantic features and spatial latent information. A dual-stream MMDiT processes the noisy visual latents together with these conditions. Figure 1 summarizes the model architecture. The separately trained Enhancer operates on generated videos (Section 2.5).

2.2 Dual-Stream MMDiT Backbone

The backbone contains 40 dual-stream Transformer [39] blocks. One stream processes visual latents, the other processes conditioning tokens, which can include text, character-related features, and reference-image semantics. Each stream has its own query, key, and value projections. The projected sequences are concatenated for joint attention, after which the outputs are split back into their respective streams. The two-stream design reflects the different origins of visual and conditioning features. Separate projections and FFNs allow each stream to transform its own representations, while joint attention allows information to pass between the streams at every layer. This design preserves modality-specific processing and information exchange throughout generation.

Timestep conditioning controls how the network processes features at different noise levels. A 256-dimensional sinusoidal timestep embedding is mapped to a 3072-dimensional representation by a two-layer MLP with an intermediate SiLU activation. We apply a SiLU activation followed by a linear projection to a 1536-dimensional bottleneck shared across all backbone blocks. Each block maintains a separate linear projection for each stream, mapping this shared representation directly to six 3072-dimensional vectors: shift, scale, and gate for the attention and FFN branches. Shift and scale modulate normalized features, while the gates control each branch's contribution to the residual update [27, 5]. The final output head uses a separate modulation path that derives shift and scale directly from the original 3072-dimensional timestep representation.

We use three-dimensional rotary position embeddings (RoPE) [36] to represent temporal, vertical, and horizontal positions. In our configuration, the 128 dimensions of each head are allocated as 16, 56, and 56 dimensions along these axes. Rotary embeddings are applied to the visual queries and keys, not to the conditioning stream. Vimoo 1.0 T2V, I2V, and Lite all use a T/H/W adaptation of YaRN [28] for training across spatial and temporal settings.

Normalization serves two distinct roles. LayerNorm [1] precedes the attention and FFN transformations, with conditional shifts and scales supplied by timestep modulation. Separately, queries and keys are normalized using RMSNorm [51] within each attention head, with distinct learned scale parameters. Controlling query and key magnitudes helps limit excessive attention-logit scales. In our training experience, QK normalization was important for stabilizing attention, including when using Muon.

2.3 Text and Image Conditioning

The conditioning and visual components are listed below. Here, "tokenizers" covers continuous encoders, adapters, and discrete tokenization.

• Text encoder. Qwen2.5-VL [2] provides semantic representations of the prompt. It then projects 3,584-dimensional text features to the backbone width of 3,072.
• Text token refiner. A two-layer module inside the denoiser adapts text tokens using self-attention and SiLU FFNs. Its conditioning combines a separate timestep embedding with projected, pooled text features. Unlike the main backbone, the refiner uses LayerNorm, gate-only modulation, and no QK normalization.
• Character representation. Glyph-ByT5-v2 [22, 45] supplies fine-grained character-related features that complement the prompt's semantic representation. Its 1,472-dimensional features are normalized and mapped to 3,072 dimensions, then supplied as a separate group of conditioning tokens.
• Reference-image semantics. SigLIP [50] encodes semantic features of the reference image. A normalization and MLP adapter maps the 1,152-dimensional interface to the conditioning width. This semantic path complements the spatial reference information carried by VAE latents.
• Visual VAE. We use the Wan 2.2 VAE variant with temporal, height, and width compression factors of 4 × 16 × 16 [41]. It encodes visual content into a compact latent space and decodes generated latents. The denoiser uses 48 latent channels.

The visual input combines 48 noisy-latent channels, 48 conditioning-latent channels, and one mask channel, giving 97 input channels. The output has 48 channels. Text, character, and image-semantic features enter the conditioning stream after their respective adapters, with embeddings identifying the condition type. Reference latents enter through the visual path, providing a separate route for spatial reference information.

2.4 Vimoo 1.0-Lite

Vimoo 1.0-Lite introduces two changes while retaining the 40-block dual-stream backbone. The first reduces the visual sequence length through spatial patchification in hidden space. After the input projection, four neighboring hidden vectors in each 2 × 2 spatial region are concatenated and linearly mapped from four times the hidden width back to the original width. The temporal axis is not merged. For an evenly divisible spatial grid, the backbone therefore processes one quarter as many visual tokens. After the backbone, an unpatchify projection expands each hidden vector to four position-specific outputs. The restored spatial layout then passes through final normalization, modulation, and the output projection. Our purpose is to use the backbone's learned capacity on a shorter sequence while retaining a fine-grid latent output. The second change further compresses timestep modulation by narrowing the shared bottleneck. Vimoo 1.0 uses a shared 3,072-to-1,536 projection, followed by 80 independent 1,536-to-18,432 output projections. The Lite model retains this structure but reduces the bottleneck width from 1,536 to 16. Both models share the bottleneck representation across all blocks and both streams, while retaining a separate output mapping and bias for each block and stream. This design builds on the shared timestep representation already used in Vimoo 1.0, exploring whether a much narrower bottleneck can provide sufficient conditioning information. For the Lite model, the backbone modulation parameters decrease from 2.271 billion to 0.025 billion. Including Lite model's added merge and split modules, the total diffusion transformer count falls from approximately 11.7 to 9.5 billion parameters.

2.5 Vimoo 1.0 Enhancer

Generative video enhancement and restoration methods aim to improve visual detail while maintaining temporal consistency [9, 43, 42].

Vimoo 1.0 Enhancer is a separately trained generative video enhancement model that follows the same overall network architecture as the Vimoo 1.0 family. It takes generated 720p video as its input and produces enhanced video at 1080p and 24 fps. It can enhance outputs from both Vimoo 1.0 and Vimoo 1.0-Lite and is also suitable for most 720p videos.

The model provides two complementary functions: 1) Generative repair. The Enhancer is designed to correct defects already present in generated content, including malformed fine details, hand and facial artifacts, ripple-like distortions during motion, and implausible motion. 2) Resolution and clarity enhancement. The Enhancer improves visual quality, sharpness, and fine-detail rendition while increasing the output resolution from 720p to 1080p. Overall, the Enhancer repairs visual artifacts and improves resolution, sharpness, and fine detail.

3.1 Training Objective and Joint Learning Tasks

We train in the visual latent space using flow matching [18]. The network predicts a conditional velocity field from the current noisy state and timestep, with text and, where applicable, reference-image information. This formulation provides a common generative objective for image and video training, while the data and conditioning determine the task presented to the model.

Our training includes T2I, mixed T2I/T2V, and mixed T2V/I2V tasks. T2I training provides an initial basis for learning subject appearance, scene structure, composition, and correspondence with text. T2V is central to the acquisition of the model's basic video-generation capabilities. Large amounts of video data are needed to learn how actions unfold, how objects interact, and how scene appearance changes with camera movement. I2V further extends generation to scenarios in which a user supplies a reference image. The model must use its appearance and content while generating motion consistent with the text condition. Joint T2V/I2V training supports both conditioning modes, making image-conditioned task organization important to training.

3.2 Progressive Training with Mixed Resolutions and Frame Rates

Prior video diffusion work has explored staged training from images to videos, followed by refinement on high-quality video data [3].

For our Vimoo 1.0 family, training starts at lower resolution with broad data coverage and progressively places greater emphasis on visual detail and quality. As training advances, data quality increases while the range and volume of selected data become more focused. This progression is intended to use computation efficiently: broad visual and semantic knowledge can be learned before concentrating more of the budget on higher-resolution refinement.

Table 1 lists the training settings. Training begins with 256p images, followed by image/video mixtures at 256p and then 480p. A mixed-setting stage combines 480p at 16 fps, 480p at 24 fps, and 720p at 16 fps. Training subsequently continues at 720p and 24 fps, followed by two SFT stages. Early stages establish broad visual representations and semantic associations, while video data supplies temporal and interaction knowledge. Higher-resolution data places greater demands on fine detail and visual consistency. In the mixed-setting stage, 480p data provides an economical setting for learning plausible dynamics, while 720p data supplies finer visual information.

A practical difficulty in mixed training is maintaining a consistent interpretation of spatial and temporal positions. In our earlier experiments, the position-extension schemes we tested could degrade generation quality when resolution or temporal settings changed. Resolution transitions sometimes required the model to relearn aspects of its prior behavior. We address this issue with a T/H/W adaptation of YaRN [28], adjusting positional representations along the temporal and both spatial axes. The intention is to let a shared backbone learn from videos with different spatial scales and temporal sampling densities. We use this approach across the whole Vimoo 1.0 family. In our training experience, this eased adaptation to changing settings and improved mixed training.

3.3 Muon and AdamW Optimization

Early optimizer comparisons motivated the use of Muon [15, 21] for large-scale training. In our setup, it improved convergence speed and helped overall training stability. We use Muon for the matrix parameters of the backbone attention and FFN layers. AdamW [23] handles the modulation layers and input/output layers, as well as non-matrix parameters such as biases and normalization scales.

Combining the optimizers requires coordinating their update scales. Their hyperparameters affect different parts of the network, so tuning either optimizer in isolation can produce imbalanced updates. We monitor parameter-update magnitudes and gradient values when adjusting the training settings. We keep this description at the level of balancing updates across parameter groups, without imposing a single update-ratio formula or a fixed relationship between their learning rates.

Faster convergence does not eliminate numerical-scale risks. During Muon training, we observed growth in some parameter norms and activation magnitudes. We therefore track parameter norms and activation RMS values across the network and adjust the training strategy when necessary. Controls on activation and parameter magnitudes, together with the QK normalization described in Section 2.2, help manage this behavior.

Last but not least, validation loss can usually guide stage progression. In addition to its aggregate trend, we monitor loss across noise timesteps and video categories to avoid relying on an average that may hide uneven learning. When validation loss stabilizes, we increase the batch size or reduce the learning rate, and continue training before moving to the next stage. Our target is sufficient convergence for that stage, rather than an assertion of mathematical optimality.

3.4 Post-Training and Model Refinement

After large-scale pretraining, we use targeted high-quality training to address capabilities that remain insufficiently learned. We first assess these areas and then adjust the data distribution and sampling mixture to increase the model's exposure to the relevant content. This uses the retrieval and mixture-organization capabilities built within our data pipeline. For broader quality refinement, we construct a more selective supervised fine-tuning (SFT) set with attention to visual quality, motion quality, caption accuracy, and cinematographic content. Training uses learning-rate annealing and maintains an exponential moving average (EMA) of model weights. Annealing reduces the update scale during refinement, while EMA smooths the sequence of model-weight updates. We also perform targeted DPO to shift generation toward the desired preferences [30, 40, 20].

3.5 Enhancer Training

We initialize the Enhancer with pretrained weights from Vimoo 1.0 to leverage its generative capabilities for video repair. Unlike text-conditioned video generation, enhancement should preserve the content of the input video rather than follow the customized text prompt. We therefore replace variable text prompts with a short, fixed sequence of text tokens throughout training and inference. We do not apply condition dropout for classifier-free guidance (CFG) during training or use CFG at inference.

The low-quality video serves as the conditioning input, using the concat injection mechanism similar to that of the I2V model. During training, the model learns to denoise a noisy version of the high-quality target video, conditioned on the corresponding low-quality input. We find that retaining a multi-step denoising process is important for effective generative repair. However, the number of denoising steps can be reduced through suitable acceleration methods.

3.6 Distributed Training and Engineering Optimization

We train the MMDiT backbone using a distributed training stack built on Megatron Core [34, 26]. The large model and long visual token sequences place high demands on computation and memory. We distribute the training workload across devices and optimize data loading and workload balancing to accommodate samples with different resolutions and lengths. These optimizations help reduce idle time caused by data loading delays and uneven workloads. We also optimize operators and execution to reduce memory traffic and training overhead.