OpenTSLM

The time series encoder is adopted among OpenTSLM’s different architecture. OpenTSLM can either train an encoder from scratch (e.g., using PatchTST architecture), or using pretrained TSFMs (e.g., Chronos-2).

When trained from scratch, the encoder is trained with the backbone models; while for pretrained encoders, their parameters are frozen, and only a projection layer is trained alongside the backbone model.

Built on Time-Series Encoder, OpenTSLM tries to leverage a pretrained LLM. It adds an extra projection layer mapping TS embeddings to LLM hidden space. Then, it finetunes the LLM with LoRA.

Then, OpenTSLM-SP integrates text and time series data with soft prompting. A typical prompt consists of

1. initial text segment (“pre-prompt”)
2. a sequence of interleaved time-series tokens and textual descriptions
3. final text segment (“post-prompt”)

Suppose the time-series tokens are \(Z_{i}\), corresponding description (text embedding) is \(T_{desc_{i}}\), then the typical model input of OpenTSLM is

\[ X_{input} = [ T_{pre}, Z_{1}, T_{desc_{1}}, Z_{2}, T_{desc_{2}}, \dots, Z_{K}, T_{desc_{K}}, T_{post} ] \]

The OpenTSLM-SP then finetunes the backbone (with LoRA) and the MLP.

Inspired by Flamingo (the VLM architecture), OpenTSLM-Flamingo uses a time-series encoder and adapt the cross-attention mechanism. The model consists of (1) time series patch encoder, as in Section 1.1; (2) Perceiver Resampler; (3) gated cross-attention layers integrated into the LLM; (4) the frozen LLM backbone.

The perceiver resampler encodes time-series patches and yields a fixed-size latent representation.

\[ Z_{latent} = \mathrm{PR}(E_{1:N}) \in \mathbb{R}^{N_{latent} \times d_{time}} \]

Then, to integrate \(Z_{latent}\) into LLM, a gated cross-attention is added every \(N\) transformer blocks which compute \(Q_{text}=xW_{Q}\) and latent as KV \(K_{ts}=Z_{latent}W_{K}, V_{ts}=Z_{latent}W_{V}\), and

\[ \mathrm{Gated-XAttn}(x,Z_{latent}) = x + \gamma \cdot \mathrm{softmax}\left( \frac{Q_{text}K_{ts}^{\top}}{\sqrt{d_{k}}} \right) V_{ts} \]

After that, following OpenFlamingo, OpenTSLM-Flamingo has to condition text-tokens on time-series via special tokens. <TS> and <endofchunk> are introduced to indicate when time-series modalities should be incorporated. Upon encountering <TS>, the model conditions on the corresponding latent representation \(Z_{latent}\) via gated cross attention.