LTX2: Efficient Joint Audio-Visual Foundation Model
LTX2 is a foundation model for text-to-audio-video (T2AV) and text-image-to-audio-video (TI2AV) with the input image being the first frame of video.
Table of Contents
1. Video and Audio Latent Space
LTX2 follows the latent diffusion architecture. It decouples the latent spaces of videos and audios. Speaking of latent spaces, it requires an encoder to map videos or audios into a latent tensor and a decoder to map the latent tensor back into videos or audios.
1.1. Video VAE
1.1.1. Spacial and Temporal Compression
Videos need to be compressed at 3 dimensions: height, width, and time1. Therefore, a video VAE encoder maps a video of shape \([B,3,T,H,W]\) into a latent tensor of shape \([B,C,t,h,w]\), where \(B\) is the batch size; \(T,H,W\) are duration (the number of frames), frame height, frame width respectively; \(C\) is the channel size in latent space (usually larger than \(3\) to capture more features); \(t,h,w\) are downsampled duration, height, and width by a certain factor such that \(t=T/r_{t}, h=H/r_{s}, w=W/r_{s}\)2
1.1.2. Casual VAE
During convolution, kernels are only allows to inspect current and prior frames, to avoid information leakage from future frames.
1.1.3. Positional Embedding for Video Latent Space
As a video has 3 dimensions, therefore, 3D RoPE is applied to \(t,h,w\) before the latent tensor is sequentialized and sent to Transformer blocks.