CLIP

Indeed, CLIP’s solution is quite simple. Given \( N \) pairs of image-text pairs, CLIP uses an image encoder to encode \( N \) images to latent representation, and uses a text encoder to encode texts to latent representation. Then, CLIP uses 2 linear projector to match the shape of image and text embeddings.

So, since we have \( N \) image embeddings and \( N \) text embeddings, we have \( N\times N \) possible predictions. CLIP learns to predict the correct pairing (by making the dot product between corresponding image-text embeddings maximized, while minimizing non-pairing embeddings).