SensorLM: Learning the Language of Wearable Sensors
Table of Contents
1. Challenges
- LLMs are unable to handle high-dimensional, continuous, and temporal extensive sensor data, which can easily exceed context length.
- Lack of large-scale and high-quality sensor-language dataset
Thus, this work contributes to
- a hierarchical automated caption-generation pipeline that systemetically captures statistical, structural, sementic features from fine-grained streaming sensor data
- the curation of a large-scale dataset
- a generic pretraining framework that integrates diverse multimodal architectures for scalable and robust learning
2. Solution
2.1. Dataset Construction
Data are collected from wrist-worn wearable sensor data, raw data include time series signals.
2.2. Hierarchical Sensor Caption Generation
Captions are generated at 3 distinct levels:
- Statistical captions.
- Provide a quantitative summary
- Statistical measures (e.g., mean, maximum, minimum, std) are computed. They are injected into templates.
- Structural captions.
- Focus on encoding the dynamic characteristics and patterns within time series sensor data, including trends, flunctuations, and other temporal features.
- Apply sliding windows to identify trends, spikes, drops. These patterns are also injected to templates.
- Semantic captions.
- Capture high-level meaning and context embedded in sensor data, which may reflect individual’s activity.
- Incorporate recognized activity and period of time; integrate user-logged mood data.
2.3. SensorLM
SensorLM uses a sensor encoder (ViT), text encoder and multimodal text decoder.
2.3.1. Model Architecture
The sensor encoder adaopts a ViT that segment the sequence into patches, apply linear embeddings and process them through Transformer blocks to local and long-range temporal dependencies. The text encoder is similar that encode text into unimodal representations. The multimodal text decoder is a casaully masked transformer that integrates both sensor embeddings (via cross-attention) and text features.
2.3.2. Pretraining Objective
SensorLM combines both contrastive (as in CLIP) and generative (as in Cap) objectives. Given a batch of \( N \) sensor-language pairs \( \{ (x_{n}, y_{n}) \}_{n\in[N]} \), the contrastive loss \( \mathcal{L}_{con} \) is applied to normalized unimodal text embeddings \( v_{i} \) from the text encoder and sensor embeddings \( s_{i} \) from the sensor encoder via a symmetric cross-modal objective
\[ \mathcal{L}_{con} = -\frac{1}{N}\left( \sum_{i=1}^{N} \log \frac{\exp(\mathrm{sim}(s_{i}, v_{i}) / \tau)}{\sum_{j=1,j\ne i}^{N} \exp(\mathrm{sim}(s_{i}, v_{j}) / \tau)} + \sum_{i=1}^{N}\log \frac{\exp(\mathrm{sim}(v_{i}, s_{i}) / \tau)}{\sum_{j=1,j\ne i}^{N} \exp(\mathrm{sim}(v_{i}, s_{j}) / \tau)} \right) \]
- The \( \mathrm{sim}(\cdot,\cdot) \) is the similarity measure between embeddings
- \( \tau \) denotes the temperature
The caption loss \( \mathcal{L}_{cap} \) is a standard cross-entropy loss applied to the outputs of multimodal text decoder.
\[ \mathcal{L}_{cap} = -\sum_{t=1}^{T} \log \mathbb{P}_{\theta}(y_{t} \mid y_{\lt t}, x) \]
The final objective is a weighted combination: \( \mathcal{L} = \lambda_{con}\cdot \mathcal{L}_{con} + \lambda_{cap}\cdot\mathcal{L}_{cap} \)