Multi-Head Attention & Multi-Head Self-Attention

MHA intends to capture information between tokens through different "aspects" with the design of "head"s, which is literally splitting the embedding dimension into several non-overlapping parts.

Attention Study/Foundation

Self Attention

Self attention is built on top of attention, meant to capture relationship between tokens in the same sequence.

Attention Study/Foundation

Naive Attention

The foundation of the whole AI.

Attention Study/Foundation