Monte Carlo CPU-GPU task scheduler

Motivation

Training a reliable speech brain-computer interface (BCI) decoder requires large amounts of stable, labeled neural data, but intracortical recordings are expensive to collect, and the signal itself drifts session to session, degrading a decoder's ability to generalize. Working with a collaborator at RWTH Aachen, we wanted to test a specific question: can synthetic spike trains, generated from a biologically grounded cortical simulation, meaningfully improve a neural decoder's generalization?

We used NEST to simulate cortical microcircuit activity that preserves the temporal structure of real spike data, then framed neural speech decoding as a sequence-to-sequence (seq2seq) problem: a GRU maps sequences of neural features to sequences of phoneme probability distributions, trained and evaluated under three data conditions (empirical only, mixed, and fully synthetic) to isolate what synthetic data actually contributes.

---

Overview

Spike2Word is an end-to-end pipeline with three stages: neural feature extraction, synthetic spike train generation, and seq2seq neural decoding.

• Synthetic spike train generation — Real threshold-crossing spike counts from one channel are converted into a time-varying firing rate, which drives an inhomogeneous Poisson generator inside a NEST simulation of a cortical microcircuit (adapted from the Potjans-Diesmann model to approximate area 6v, the speech-motor region the real data comes from).
• Label-preserving data augmentation — Critically, the sentence label from each real trial is kept fixed while the neural input sequence is regenerated through simulation. This means synthetic and empirical trials can be mixed directly during training without needing new transcriptions.
• GRU-based seq2seq decoder — A 5-layer bidirectional GRU, trained with CTC loss, maps binned neural feature sequences to phoneme probability distributions, decoded via greedy CTC decoding into a final phoneme sequence.
• Three-condition evaluation — Identical decoders are trained on empirical-only, mixed empirical + synthetic, and fully synthetic data, then compared on CTC loss and character error rate (CER) to isolate what synthetic data changes.

---

Data

We use the same public dataset from Willett et al. (2023) as our starting point: intracortical recordings from a human participant attempting to speak 10,850 sentences, structured as competitionData with tx1–tx4 threshold-crossing features (20ms bins, 256 channels, thresholds from −3.5× to −6.5× RMS) and spikePow spike-band power features per channel.

From real data to synthetic spike trains. For each 20ms bin i with spike count xᵢ, we estimate a firing rate λ̂ᵢ = xᵢ / Δt, then build a piecewise-constant firing rate function λ(t) over the whole utterance. This drives NEST's inhomogeneous Poisson generator, whose output feeds the Potjans-Diesmann cortical circuit simulation.

From NEST's raw output to model input. NEST outputs raw spike-time sequences, each spike a (NeuronID, time_ms) pair. We bin these at 20ms resolution across 128 electrodes, producing a spike-count matrix of shape T × 128 per simulated trial, the same shape and resolution as the real tx1 features, which is what makes empirical and synthetic trials directly interchangeable during training.

We validated this conversion qualitatively before training anything: a heatmap of real tx1 threshold crossings confirmed every channel carried distinguishable signal, and a raster plot of simulated spikes (grouped by cortical layer L2/3, L4, L5, L6) showed high-activity periods in the synthetic data lining up with the real data's activity envelope, confirming the simulation preserves genuine temporal structure rather than producing generic noise.

---

Method

Real tx1 spike counts (20ms bins, per channel)
        │
        ▼
Estimate firing rate λ(t)  ──►  NEST inhomogeneous Poisson generator
        │
        ▼
NEST cortical microcircuit simulation (Potjans–Diesmann, adapted to area 6v)
        │  500ms initialization → stimulus applied → spikes recorded across L2/3, L4, L5, L6
        ▼
Bin spike times at 20ms  ──►  spike-count matrix (T × 128), label-preserving
        │
        ▼
Gaussian smoothing (low-pass filter on neural signal)
        │
        ▼
Day-specific affine transform + Softsign nonlinearity (normalizes across recording sessions)
        │
        ▼
5-layer bidirectional GRU  ──►  per-timestep phoneme logits (+ CTC blank token)
        │
        ▼
CTC loss (training)  /  Greedy CTC decoding (inference)
        │
        ▼
Predicted phoneme sequence  ──►  compared against ground-truth phonemes

Preprocessing layers before the GRU. Two preprocessing steps run on the binned neural sequences before the recurrent model sees them: a Gaussian smoothing layer acts as a low-pass filter, reducing high-frequency noise while preserving overall temporal structure; then a day-specific affine transform followed by a Softsign nonlinearity normalizes neural feature distributions across recording sessions, directly addressing electrode drift and day-to-day variability, one of the central problems the project set out to address.

Why CTC instead of a fixed alignment. Neural feature sequences and phoneme label sequences don't have a known one-to-one alignment, so CTC loss lets the model learn its own alignment between input time steps and output phonemes, including a blank token that allows flexible many-to-one mappings. At inference, we use greedy decoding: take the most probable token at each time step, then collapse repeated tokens and blanks into the final phoneme sequence.

Why GRU over an attention-based architecture. Seq2seq models built on gated recurrent units impose strong temporal biases suited to continuous, noisy input, and are more robust than attention-based architectures in the low-data regimes common in BCI work, which matched our constraints well given the dataset size we were able to work with (see Limitations).

---

Experimental settings

All experiments use a custom neural sequence decoder implemented in PyTorch, trained on the GPU partition using NVIDIA A100 GPUs (40GB memory, two GPUs per node, full node memory requested via --mem=0). A single training run took several hours to complete.

Training hyperparameters (best-performing configuration found during development):

We conducted targeted experiments over learning rate, GRU hidden size, number of recurrent layers, dropout, and temporal preprocessing settings before settling on this configuration.

---

Results

Quantitative

The empirical-only and mixed conditions produced identical loss and CER, indicating that at this experiment's scale, adding synthetic samples to real data neither helped nor hurt decoding accuracy. The fully synthetic condition showed a lower training loss (5.24) but a marginally higher CER (0.80), an apparent mismatch that makes sense given what each metric measures: CTC loss reflects how much probability mass the model assigns to the correct label sequence, while CER measures the actual decoded output after greedy inference, so a better-calibrated model (lower loss) can still make a few more discrete errors after decoding.

Qualitative

Five example predictions against ground truth, generated via greedy CTC decoding:

The model captures a rough phonetic skeleton of each sentence, but as the loss/CER numbers suggest, exact recovery is far from reliable, consistent with training on a small, constrained dataset (see Limitations).

---

Analysis and discussion

Our hypothesis was that synthetic neural data could improve BCI decoder generalization. The results partially support that: the mixed-data condition showed no measurable CER improvement over empirical-only training, suggesting that, as generated in this study, synthetic data may not introduce enough variability beyond what's already present in the real recordings. But the fully-synthetic condition's lower training loss suggests the simulated data does capture temporally relevant structure that could be a useful signal, particularly in low-data regimes, even if that hasn't translated into a CER improvement yet.

Overall: synthetic spike data appears to capture real temporal structure relevant to neural speech decoding, but whether it actually helps depends on scale, diversity, and how it's integrated into training. These are preliminary findings, not a settled result.

---

Limitations

The biggest constraint on this project was computational and operational scalability, not modeling choices. The Potjans-Diesmann model simulates roughly 80,000 neurons and 300 million synapses at 0.1ms resolution, making each simulation run highly compute-intensive. Because we couldn't build a fully automated, sentence-level simulation pipeline, the workflow stayed largely manual, which forced us to restrict the training set to just fifty sentences, well below what's needed to reliably train and evaluate a GRU decoder. With that few examples, the model is prone to high-variance estimates and metric sensitivity to small changes in decoding, so every trend reported above should be read as preliminary rather than conclusive.

We also weren't able to train a language model to convert phoneme probability distributions into full words, the n-gram model we initially planned to use had memory requirements that made it infeasible at our scale. Evaluation was limited to CER, training loss, and qualitative phoneme-vs-ground-truth comparisons; we didn't measure Phoneme Error Rate or run explicit cross-session generalization tests, both of which would likely reveal augmentation benefits (or lack thereof) that CER alone doesn't capture.

---

What's next

• Scaling past fifty sentences — building a faster, more automated simulation pipeline to make larger training sets computationally feasible.
• Explicit cross-day evaluation — directly testing whether synthetic augmentation improves robustness to session-to-session neural drift, the original motivating problem.
• Pretrain-then-finetune — pretraining on synthetic data and finetuning on empirical recordings, rather than training on a fixed mixture, to see if that ordering surfaces benefits the current setup doesn't.
• A working phoneme-to-word language model — revisiting the n-gram approach or a more memory-efficient alternative to complete the phoneme-to-text stage of the pipeline.
• Additional metrics — adding Phoneme Error Rate alongside CER for a fuller picture of decoding quality.

---

Resources

• GitHub Repo
• Full Report