[2102.09532] Clockwork Variational Autoencoders for Video Prediction

Deep learning has enabled algorithms to generate realistic images. However, accurately predicting long video sequences requires understanding long-term dependencies and remains an open challenge. While existing video prediction models succeed at generating sharp images, they tend to fail at accurately predicting far into the future. We introduce the Clockwork VAE (CW-VAE), a video prediction model that leverages a hierarchy of latent sequences, where higher levels tick at slower intervals. We demonstrate the benefits of both hierarchical latents and temporal abstraction on 4 diverse video prediction datasets with sequences of up to 1000 frames, where CW-VAE outperforms top video prediction models. Additionally, we propose a Minecraft benchmark for long-term video prediction. We conduct several experiments to gain insights into CW-VAE and confirm that slower levels learn to represent objects that change more slowly in the video, and faster levels learn to represent faster objects.

3 mentions: @ak92501@evolvingstuff@arxivabs
Keywords: autoencoder
Date: 2021/02/20 18:53

Referring Tweets

@evolvingstuff Clockwork Variational Autoencoders for Video Prediction "We introduce the Clockwork VAE (CW-VAE), a video prediction model that leverages a hierarchy of latent sequences, where higher levels tick at slower intervals" t.co/tZPubGIZO8 t.co/D7ayYt6OJB
@arxivabs @danijarh @saxenavaibhav11 Check out the abstract! t.co/JzWS5a5jlh

Related Entries

Read more [2012.11995] Pre-Training a Language Model Without Human Language
0 users, 3 mentions 2020/12/23 03:51
Read more [2102.03161] PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformers
0 users, 2 mentions 2021/02/08 02:21
Read more [2102.02888] 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Sp...
0 users, 2 mentions 2021/02/08 02:21
Read more [2102.06431] VARA-TTS: Non-Autoregressive Text-to-Speech Synthesis based on Very Deep VAE with Resid...
0 users, 2 mentions 2021/02/15 02:21
Read more [2102.07492] DOBF: A Deobfuscation Pre-Training Objective for Programming Languages
0 users, 3 mentions 2021/02/16 14:21