Abstract
V-RAE constructs semantically organized video latents from frozen vision representations to improve generation quality, convergence speed, and predictive modeling.
Latent video generation relies on autoencoders to define a compact space in which generative models operate. Although video autoencoder architectures have evolved substantially, their latent spaces are still optimized primarily for pixel-level reconstruction and provide limited high-level semantic organization. A reconstruction-optimal latent space, however, need not be well suited to generative modeling. We propose V-RAE, a video representation autoencoder that builds compact generative latents on top of frozen vision foundation model representations. A lightweight temporal pooling module removes temporal redundancy while preserving semantic structure, and a video decoder reconstructs continuous motion from the compressed features. We evaluate V-RAE with four representative frozen encoders on video reconstruction, semantic probing, and class-conditional generation. V-RAE achieves 2.13 rFVD on K600, outperforming all evaluated large-scale pretrained video VAEs. Its latents retain substantially more semantic information than conventional video tokenizer latents. Under matched generation settings, our best variant achieves gFVD scores of 117.86 and 19.16 on UCF101 and K600, respectively, while converging up to 6x faster}. We further show that reconstruction quality alone is insufficient to characterize generative utility and introduce tFVD, a temporal-coherence diagnostic that correlates more reliably with downstream generation quality. Beyond video generation, V-RAE also improves future video prediction on Cityscapes over the Wan 2.2 VAE latent space under matched prediction settings. Taken together, the experiments show that frozen semantic representations can support video reconstruction, generation, and predictive modeling. The project page: https://v-rae.github.io/.
Community
V-RAE: Extending Representation Autoencoders to Video Generation
Following the recent Representation Autoencoder (RAE) paradigm in image generation, we explore a natural question:
Can pretrained visual representations also serve as effective latent spaces for video generation?
Our answer is yes — but extending RAE from images to videos introduces several new challenges and, more importantly, reveals new properties of generative latent spaces.
We introduce V-RAE (Video Representation Autoencoder), extending representation-based latent modeling from images to videos by transforming pretrained visual features into compact and temporally coherent video latents.
Key highlights:
🧠 From Image RAE to Video RAE.
V-RAE extends the RAE paradigm to video generation, leveraging pretrained visual encoders such as DINOv3, SigLIP2, EUPE, and V-JEPA 2.1 instead of learning reconstruction-oriented latents from scratch.⏱️ Video requires temporal compression.
Unlike images, video representations contain substantial temporal redundancy. V-RAE introduces temporal aggregation to compress frame-level representations while preserving semantic and motion information.🎯 Semantic latents remain highly effective for video generation.
V-RAE preserves substantially richer semantic information than conventional video VAE latents and consistently improves downstream generation quality under matched generator settings.⚡ Up to 6× faster convergence.
Generative models learn V-RAE latents significantly faster, showing that semantically structured representations can make the video generation problem substantially easier to learn.🔄 Good reconstruction ≠ good generation.
A particularly important observation in video is that reconstruction quality does not reliably predict downstream generation quality. A latent space with lower rFVD can still be harder for a generator to model.📏 tFVD better predicts generative utility.
Motivated by this mismatch, we introduce tFVD, which captures temporal properties of the latent space and correlates substantially better with downstream generation performance than reconstruction FVD.🔮 Beyond generation: future prediction.
The advantages of V-RAE also transfer to future video prediction, suggesting that representation-based video latents are useful beyond standard video generation.
Takeaway:
V-RAE shows that the RAE paradigm extends naturally from images to videos, while revealing a video-specific insight: a good generative latent space should preserve semantic and temporal structure, not merely optimize reconstruction quality.
Get this paper in your agent:
hf papers read 2608.13556 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 1
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper