TimeXer
Table of Contents
1. Problem Settings
Suppose endogenous time series \( \mathbf{x}_{1:T} = \set{\mathbf{x}_{1}, \mathbf{x}_{2},\dots,\mathbf{x}_{T}} \) where \( \mathbf{x}_{i} \) denotes the value at \(i\)-th time point and \(T\) is the look back window length of the endogenous variable.
We also have multiple exogenous series \( \mathbf{z}_{1:T_{ex}} = \left\{ \mathbf{z}_{1:T_{ex}}^{(1)}, \mathbf{z}_{1:T_{ex}}^{(2)}, \dots, \mathbf{z}_{1:T_{ex}}^{(C)} \right\} \in\mathbb{R}^{T_{ex} \times C} \), where \( C \) is the number of exogenous variables, \( T_{ex} \) is the look back window length for exogenous variables and \( \mathbf{z}^{(i)}_{1:T_{ex}} \) represents the \(i\)-th exogenous variables from time point \(1\) to \(T_{ex}\).
The goal of a forecasting model \(\mathcal{F}_{\theta}\) is to predict the future \(S\) time steps \( \widehat{\mathbf{x}} = \left\{ \mathbf{x}_{T+1}, \mathbf{x}_{T+2}, \dots, \mathbf{x}_{T+S} \right\} \) based on historical observations and corresponding exogenous series:
\[ \widehat{\mathbf{x}}_{T+1:T+S} = \mathcal{F}_{\theta}(\mathbf{x}_{1:T}, \mathbf{z}_{1:T_{ex}}) \]
2. TimeXer Solution
TimeXer utilizes Transformers and uses self-attention and cross-attention to capture temporal-wise and variate-wise dependencies.
2.1. Endogenous Embedding
TimeXer uses non-overlapping patch-wise representation.
Given the distinct roles of endogenous and exogenous variables, TimeXer embeds the former at smaller granularity (patch-wise) while embeds the latter with larger granularity (variate-wise). To avoid information misalignment between exogenous variable embedding and endogenous variable embedding, TimeXer introduces learnable global token for endogenous variables. The procedure is shown below:
\begin{equation} \begin{split} \set{\mathbf{s}_{1}, \mathbf{s}_{2},\dots, \mathbf{s}_{N}} &= \text{Patchify}(\mathbf{x}) \\ \mathbf{P}_{en} &= \text{PatchEmbed}(\mathbf{s}_{1}, \mathbf{s}_{2},\dots, \mathbf{s}_{N}) \\ \mathbf{G}_{en} &= \text{Learnable}(\mathbf{x}) \end{split} \end{equation}Suppose \(P\) is patch length and \(N = \lfloor \frac{T}{P} \rfloor\) number of patches. \(\text{PatchEmbed}(\cdot)\) maps each patch and positional embedding into a \(D\)-dimensional vector via a trainable linear projector. After the procedure, \(\mathbf{P}_{en}\) is a sequence of \(N\) patch-level temporal tokens and \(\mathbf{G}_{en}\) is the global token for that endogenous variable.
2.2. Exogenous Embedding
Interaction of different variables can be captured more naturally by variate-level representations, which are adaptive to common issues, see Alternative Embedding Approach.
\[ \mathbf{V}_{ex,i} = \mathrm{VariateEmbed}\left(\mathbf{z}^{(i)}\right), \ i \in \set{1,\dots,C} \]
Here, \(\mathrm{VariateEmbed}:\mathbb{R}^{T_{ex}} \mapsto \mathbb{R}^{D}\) is a trainable linear projector.
2.3. Inter- and Intra-Attention
TimeXer developed distinct attention mechanism for learning endogenous and exogenous relationships.
2.3.1. Endogenous Self-Attention
TimeXer learns the relationships between endogenous variables through 3 approaches:
- patch-to-patch self-attention. to capture information within series
- global-to-patch cross-attention. to aggregate patch-level information across entire series
- patch-to-global cross-attention. to receive variate-level correlation
And it happens that, we can use one single self-attention to accomplish the above 3 operations with tensor concatenation.
\[ \widehat{\mathbf{P}}_{en}^{\ell}, \widehat{\mathbf{G}}_{en}^{\ell} = \mathrm{LayerNorm}\left( \left[ \mathbf{P}^{\ell}_{en}, \mathbf{G}_{en}^{\ell} \right] + \mathrm{SelfAttn}\left( \left[ \mathbf{P}^{\ell}_{en}, \mathbf{G}_{en}^{\ell} \right] \right) \right) \]
Here, \(\ell\in\set{0,\cdots, L-1}\) denotes the \(\ell\)-th TimeXer block and \(\mathbf{P}^{0}_{en}=\mathbf{P}_{en}, \mathbf{G}^{0}_{en} = \mathbf{G}_{en}\). \([ ]\) denotes tensor concatenation along sequence dimension.
2.3.2. Exogenous-Endogenous Cross-Attention
Cross-attention is popular in multi-modal learning to capture token-wise dependencies between different modailties.
Cross-attention layers in TimeXer uses endogenous global token as query and exogenous embedding token as KV.
\begin{equation} \label{orgbd29683} \widehat{\mathbf{G}}_{en, upd}^{\ell} = \mathrm{LayerNorm}(\widehat{\mathbf{G}}_{en}^{\ell} + \mathrm{CrossAttn}(\widehat{\mathbf{G}}_{en}^{\ell}, \mathbf{V}_{ex})) \end{equation}Then, we’ll transform all temporal tokens \(\mathbf{P}_{en}^{\ell}\) and learnable global tokens \(\mathbf{G}_{en}^{\ell}\) with FFN (linear).
\begin{equation} \label{org731cfc7} \begin{split} \mathbf{P}^{\ell+1}_{en} &= \mathrm{FFN}(\widehat{\mathbf{P}}^{\ell}_{en}) \\ \mathbf{G}_{en}^{\ell+1} &= \mathrm{FFN}(\widehat{\mathbf{G}}^{\ell}_{en}) \end{split} \end{equation}We write the above Transformer block [BROKEN LINK: eqref:crossattn] and [BROKEN LINK: eqref:crossattn-ffn] as \( \mathbf{P}_{en}^{\ell+1}, \mathbf{G}_{en}^{\ell+1} = \mathrm{TrmBlock}(\mathbf{P}_{en}^{\ell}, \mathbf{G}^{\ell}_{en}) \)
2.4. Forecasting Loss
Since exogenous variables are not required to be predicted, TimeXer directly applies a linear projector on the output endogenous embedding of last layer \( [\mathbf{P}_{en}^{L}, \mathbf{G}_{en}^{L}] \). The loss is measured by L2 squared loss between prediction and ground truth.
\[ \text{Loss} = \sum_{i=1}^{S} \left\| \mathbf{x}_{i} - \widehat{\mathbf{x}}_{i} \right\|^{2}, \text{ where } \widehat{\mathbf{x}}_{i} = \mathrm{Projection}([ \mathbf{P}_{en}^{L}, \mathbf{G}_{en}^{L} ]) \]
2.5. Parallel Multivariate Forecasting
Multivariate forecasting can be viewed as predicting each variable, with all other variables treated as exogenous variables. So we can assume the independence, and apply TimeXer multiple times for multivariate prediction.