Vanilla Attention
Table of Contents
1. Vanilla Attention
The idea of attention is similar to the procedure of looking up a word in the dictionary. Suppose we want to look up for the word xxxxness and we have already known the meaning of xxxx. Then we may assume that the meaning of xxxxness has something to do with xxxx.
The concept of “has something to do with” quantifies with the help of vector math. Given a token (i.e., word) of query \(q\), we have a dictionary consisting of keys \(K\) and values \(V\) (the values are vectors called word embedding which capture the meaning of tokens). We can interpret the query token with our dictionary by linear interpolation, based on similarity (aka. attention score) between query and keys
\[ A=\sum_i \texttt{sim}(q,k_i)v_i \]
So the question becomes: how to compute the similarity between query and keys, reasonably?
1.1. Scaled Dot-Product Attention
As introduced in the pioneering Transformer paper, we use softmax over cosine similarity between vectors as attention score. The cosine similarity is further extended to matmul for batch processing queries.
\[ A=\texttt{softmax}(\frac{QK^\top}{\sqrt{d}})V \]
One thing to notice is that denominator \(\sqrt{d}\). \(d\) stands for the dimension of embedding. If we assume all matrices following distribution \(\mathcal{N}(0,1)\) (to prevent from gradient explosion), then their dot product have a variance of \(d\), thus the denominator reduces the variance to \(1\) again.