LoRA

In LoRA paper’s analysis, weight matrices of pretrained LLMs have low intrinsic dimensions and hypothesize that weight update matrices also have low intrinsic dimensions, i.e.,

\[ \Delta W=BA\in\mathbb{R}^{d\times k} \]

where \(B\in\mathbb{R}^{d\times r}, A\in\mathbb{R}^{r\times k}\) and low dimension \(r\lll\min(d,k)\). Suppose the input is \(x\), then the forward propagation is

\[ \text{activation}=W'x=Wx+BAx \]

During finetuning, we fix the pretrained param \(W\), and only finetune \(B,A\). For initialization, we perform random Gaussian for \(A\) and zero for \(B\). In addition, we scale \(\Delta W\) by \(\alpha/r\), where \(\alpha\) is a fixed hyper-parameter.

LoRA does not introduce inference latency. Suppose we want to apply our finetuned model, after training \(A,B\), we simply add their product to the weight matrix:

\[ W'\gets W+BA \]

If we want to switch to a different task, we first remove the older \(A,B\) and apply the newer \(A,B\)

\[ W'\gets W-BA+B'A' \]