Common Metrics
Table of Contents
1. Text Generation
- Loss (NLL, Negative Log Likelihood)
For a language model, suppose the reference sequence is \(x_{1}, x_{2}, \dots, x_{T}\), the negative log likelihood is defined as \[ \text{NLL} = -\frac{1}{T}\sum_{t=1}^{T} \log p_{\theta}(x_{t} \mid x_{\lt t}) \]
Intuitively, closer the loss is to \(0\), closer what the model outputs is to the correct answer (i.e., reference sequence).
- Perplexity (ppl)
- Perplexity is simple \(\text{PPL} = e^{\text{NLL}}\). Lower perplexity means that the model assigns higher probability to the observed text. However, lower perplexity cannot conclude a good reasoning ability of the model, nor instruction following, nor conversation.
- BLEU
- ROUGE
- BERTScore (Precision, Recall, F1)
BERTScore is a metric for evluating generated text by comparing it with a reference using contextual token embeddings (usually BERT or RoBERTa). To some extent, it measures semantic or textual similarity, not factual or logical correctness.
Compared to BLEU and ROUGE who count “how many words overlap”, BERTScore tries to answer “how semantically similar are the words in these 2 sentences”.
BERTScore has 3 parts, suppose the reference tokens are \(\mathbf{x}_{i}, i \in [1,m]\), while the output tokens are \(\mathbf{y}_{j}, j \in [1, n]\). The similarity is often computed by cosine similarity, denoted \( \cos(\mathbf{x}_{i}, \mathbf{y}_{j}) \)
- Precision. For each output token, find the most semantically similar reference token and take average. \[ P_{\text{BERT}} = \frac{1}{n} \sum_{j=1}^{n} \max_{i} \cos(\mathbf{y}_{j}, \mathbf{x}_{i}) \]
- Recall. For each refernce token, find the most semantically similar output token and take average. \[ R_{\text{BERT}} = \frac{1}{m} \sum_{i=1}^{m} \max_{j} \cos(\mathbf{x}_{i}, \mathbf{y}_{j}) \]
- F1, which is the commonly reported “BERTScore”. \[ F_{\text{BERT}} =2\cdot \frac{P_{\text{BERT}}R_{\text{BERT}}}{P_{\text{BERT}} + R_{\text{BERT}}} \]