in-progress

Hybrid State-Space Attention for Multi-Task Vibrational Spectroscopy

Tubhyam Karthikeyan

In preparation

Architecture

Spekron uses a multi-stage encoder:

  1. Wavelet Embedding: Daubechies-4 DWT decomposes spectra into multi-scale coefficients, followed by 1D CNN patching and wavenumber-aware positional encoding.

  2. Mamba Backbone: 4 selective SSM blocks process the sequence with O(n) complexity, capturing long-range spectral dependencies.

  3. Mixture of Experts: Top-2 gating routes tokens to specialized expert networks, with optional KAN activations.

  4. Transformer Encoder: 2 global attention blocks with 8 heads for cross-position reasoning.

  5. VIB Head: Variational Information Bottleneck splits the latent into z_chem (transferable chemistry) and z_inst (discardable instrument signature).

Results

  • Pretraining loss: 1.29 → 0.90 in first 20 steps (corpus of 61K spectra)
  • Classical baseline comparison: Direct Standardization R² = 0.69 on corn moisture
  • Target: R² > 0.95 with ≤10 transfer samples (outperforming LoRA-CT)