SigLIP: Sigmoid Loss for Language Image Pretraining
Table of Contents
The advantage of SigLIP over CLIP is that SigLIP only relies on image-text pairs rather than global pairwise similarity, reducing computational cost from quadric to linear w.r.t. batch size.
1. Solution
1.1. Naive Approach
Given dataset \( B = \{ (I_{i}, T_{i}) \}_{i=1}^{N} \), the loss is defined as
\[ -\frac{1}{N} \sum_{i=1}^{N} \sum_{j=1}^{N} \underbrace{\log \frac{1}{1 + e^{ z_{ij} (-t \mathbf{x}_{i}\cdot\mathbf{y}_{i} + b)} }}_{\mathcal{L}_{ij}} \]
Here, the \(z_{ij}\) refers to binary classification, if image \(i\) is paired with text \(j\), then \(z_{ij}=1\); \(-1\) otherwise. To avoid the problem that too many negative losses may dominate the loss and cause more training steps, a bias \(b\) is introduced besides temperature \(t\).
Initially, \(t_{0} = \log 10\) and \(b_{0} = -10\)
1.2. Optimization
The above computation still requires \(O(N^{2})\) time complexity for cosine similarities. However, we notice that the computation of cosine similarity does not depend on the results of other cosine similarities, compared to softmax approaches. This means that, we can parallelize the computation. In the SigLIP paper, the parallelization is just simple chunkification into blocks.